Motivation and enhancement of physical and mental exercise, rehabilitation, health and social interaction

ABSTRACT

The disclosed invention primarily relates to methods for assisting and motivating persons with respect to various exercise and rehabilitation regimens they might undertake, mentally as well as physically. The invention also has potential application to for diagnosis and/or treatment of certain mental and physical disorders, and in other situations where a form of companionship may be provided the user. In addition the invention herein provides an enjoyable means of social interaction with others providing further motivation for physical and mental activity represented.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.12/358,404, filed Jan. 23, 2009, which claims the benefit of U.S.provisional application 61/022,991, filed Jan. 23, 2008, entitled“Motivation and Enhancement of Physical and Mental Exercise,Rehabilitation and Health”; which herein is incorporated by reference.

The application herein is a continuation of U.S. application Ser. No.12/358,404, which is a continuation in part of U.S. Ser. No. 11/980,717“Camera Based Video Games and Related Methods for Exercise Motivation”filed Oct. 31, 2007; and of U.S. Ser. No. 11/832,134 filed Aug. 1, 2007entitled Reconfigurable Tactile Control Display Applications. Ser. No.11/980,717 is a continuation of Ser. No. 11/118,774 filed May 2, 2005,now U.S. Pat. No. 7,328,119, which is a continuation in part of Ser. No.09/799,797 filed Mar. 7, 2001, now abandoned. Benefit to all of theseearlier applications is claimed, and they are hereby incorporated byreference.

The invention is related to several of my previous and in some cases copending applications, such as Ser. No. 10/934,762; Ser. No. 11/429,305;U.S. Pat. No. 6,766,036; U.S. Pat. No. 6,750,848; U.S. Pat. No.7,015,950; U.S. Ser. No. 11/186,898 (App. Pub No. 20060033713), thedisclosures of which are incorporated by reference herein. Thesedisclosures illustrate method and apparatus for determining features onobjects and persons which can be used for entertainment and gaming, aswell as more serious purposes such as weight loss, physicalrehabilitation, diagnosis of illnesses and the like.

FIELD OF THE INVENTION

The invention is generally in the field of exercise related equipmentand systems, particularly those using at least in part, one or moreelectro-optical sensors such as a TV camera to determine actions of auser and/or equipment such as exercise machines which in turn maycontrol or otherwise be inputted to a program for the purposes ofgaming, physical rehabilitation, assessment, diagnostics, training,health maintenance purposes and the like. The principal applications areentertainment and physical and/or mental health related, separately orin combination.

BACKGROUND OF THE INVENTION

A well known health care problem relates to the lack of motivation ofpersons wishing, or needing, to undertake exercise or rehabilitationtherapy. They therefore do not undertake the desired treatment steps tomaintain wellness or to return to the workplace. The result is anenormous burden on the public as well as on the persons own well being.For example the Governor of Michigan in November 2007 in a speech to theGovernors Fitness Council obesity related issues alone were costing thestate 12 billion dollars per year. This is due to direct medical costsand lost work time. Additional costs also relate to persons undertakingsuch activity in a manner that does not achieve the desired effect. Forexample, short changing a regimen prescribed to make ones wrist healfrom carpal tunnel damage. Doing ones exercises right is a big problemtoo, and recovery can be impeded if not done well.

Another aspect is mental health, which often can be present along withphysical health problems or lead to same. Physical activity can behelpful for mentally ill persons, especially those on antipsychoticdrugs that cause weight gain. Here too a motivation is needed, indeeddoubly so one might argue.

Finally because of the popularity of video games, it may be possible touse these not only to provide the motivation, but also to provide themeans to take data which can be used to guide therapy and providediagnosis.

Several of these concepts have been disclosed in my related co-pendingapplications. Recently Nintendo company has been extremely successful incommercializing a very nicely thought out device, the Wii, that takesone step toward providing the requisite motivation. The Wii however,lacks a crucial element, namely the ability to see points on you, or anexercise machine or object you are working with. In addition, it can betricked by using small movements to stand in for big ones. Since itdoesn't in many cases monitor position or motion directly, it can'tadvise you in real time or otherwise if your movement is correct, orneeds correction. And it cannot accumulate data on your activity thatwould be desirable for medical or other purposes.

SUMMARY OF THE INVENTION

The invention herein discloses simple, robust and affordable methods andapparatus to motivate people to do exercise either on exercise machinesor in free space, while accumulating data on their movements as desiredfor medical or other purposes. One or more 2-D or 3-D cameras obtaindata from various portions of a person and/or exercise machine or otherexercise related object. This data relating to positions orientationsand movements of these portions is in a preferred embodiment used as aninput to a video game, replacing in whole or in part, the input controlsof the standard button type or other game controller for which the gamewas initially designed. The player then can by moving his or her limbs,or a machine apparatus, engage in exercise while playing a game thatwould be of interest. This then provides motivation to undertake theexercise, and special game programs or subprograms for this purpose, orfor diagnostic purposes may also be created as will be disclosed.

Another purpose of the invention is to enable the simultaneousaccumulation of data concerning the exercise or therapeutic movementactivity. The data may also be used to create the enhanced gaminggraphics or other gaming and social activities and opportunities offeredby the invention and for potentially diagnosing of both mental andphysical disorders such as movement disorders and social disorders. Athird and related aspect of the invention is to provide newopportunities for therapy for such disorders, and in a way that couldincrease the chances of the person undertaking therapy.

It is also a goal of the invention to provide method and apparatus toenable persons to undertake further and better exercise than theynormally would, as a result of increased motivation and other factors.This is particularly the case with exercise involving other wise boringexercise machines which however are often the very ones needed to allowserious weight loss and/or rehabilitation of injuries. At the same timethe invention allows them to undertake such activity with reduced riskof overexertion or strain.

It is a goal of the invention to provide improved methods of physical oroccupational rehabilitation therapy, while at the same time increasingthe effectiveness of home treatment and providing information forphysical or occupational therapists to indicate movement in a givenangle or dimension, the number of repetitions and rate achieved by aconsumer, and the progress made while further motivating and informingthe consumer.

It is a goal of the invention to provide low cost computer andelectrooptical means for determining positions and movements of personsand machines at multiple points at rates sufficient to determine theinformation desired for a particular game or exercise, and to displayneeded information, goals, historical data and other information of useto the person on a display, which in many cases may already be presentfor TV watching or the like.

It is a goal of the invention to provide a method and apparatus useableon a plurality of different types of exercise machines in order to forexample allow the same game to be played using each machine, but perhapsusing different machine features or human portions as the input to thegame

It is a further goal of the invention to provide a means for disabledpersons, including those in wheelchairs, to play entertaining gameswhich further may provide helpful exercise of upper body and otherportions.

It is an additional goal of the invention to provide methods to assistin the treatment of mental, behavioral, or other disorders and insituations where a form of companionship may be provided the user,including remote real or virtual companions using the Internet or othermeans.

It is a goal of the invention to provide a method for directlydetermining a persons positions, orientations or movements for exerciserelated activity which allows limits or guidelines to be set which canbe accurately monitored to see if criteria are achieved.

And it is also a goal of the invention to provide a method for directlydetermining a persons positions, orientations or movements for exerciserelated activity which allows limits or guidelines to be set which canbe accurately monitored to see if criteria are achieved or for internetmulti-person activity especially in addition it makes up for a generaldeficiency of Nintendo Wii which doesn't give actual positions of aperson and thus while encouraging exercise, it doesn't really answerneeds of many who need help.

It is a further goal of the invention to assist a user to optimize theirperformance of exercise or other activity using data feedback frompoints determined by the invention, as well as optional information suchas heart rate, oxygen, etc.

It is another goal of the invention to provide a method and apparatususeable on a plurality of exercise machines simultaneously for bothsensing of persons and objects used as well as to have multiple persongames.

It is also a goal of the invention to provide method and apparatusutilizing natural or artificial features on a person or their clothing,to enable their positions and movements to be entered into the computerfrom which their activity may be calculated

It is also a goal of the invention to provide method and apparatus forproviding datum's on exercise machines to enable their positions andmovements to be entered into the computer, to calculate activity of aperson using them.

It is a further goal of the invention to provide method and apparatusfor determining motion of persons and calculate caloric burn rate therefrom, which data can be transmitted over the internet to healthproviders or others if desired, along with other data such as degree ofarm extension, ankle rotation and the like.

It is a goal of the invention to provide for means to control an avatarof themselves or someone in a social networking site or other venue andwhere desired to incorporate this function into a game involvingexercise.

It is a goal of the invention to provide life size stand up screen foruse when exercising to visualize computer graphics, or others sociallyinteracting with you. The screen may also be used for normal HDTV byturning it sideways.

It is a goal of the invention to provide games involving ordinaryexercise machines such as bicycles, treadmills, elliptical machines andthe like which can both monitor machine locations as well as locationson the person using them in order to effect the game.

It is also a goal of the invention to predict the effect on onesappearance (clothed, or even unclothed) of future activity such as dietor exercise and to allow one to iterate the prediction, such that a usercan optimally tradeoff, the various factors such as caloric intake,exercise time and the like, with the appearance predicted to result fromsame.

It is a goal of the invention to affect a method for social interactionfor example over the internet while undertaking games of the invention.

It is another goal the invention to provide simple means by whichconventional game consoles or game computers can be adapted for use withthe invention herein.

It is also a goal to provide simplified machine vision processing meanswhich can be optimized for the game to be played and/or the exercisemachine to be used.

It is a further goal of the invention to provide methods for calibratingsystems with respect to the sensor and display device, in considerationof the game or other program requirements.

It is another goal of the invention to provide simplified methods ofsetting up the system with different exercise equipment and in differentlocales.

Further features and advantages of the present invention will be setforth in, or apparent from, the detailed description of preferredembodiments thereof which follows.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 a illustrates a top view of an exercise bicycle embodiment.

FIG. 1 b illustrates a side view of the bicycle of FIG. 1 a, furtherincluding a person (also called rider, or player in this document)riding the bike.

FIG. 1 c illustrates simplified image processing aspects.

FIG. 1 d illustrates calibration also as a function of a game.

FIG. 2 a illustrates one method of utilizing commercial video gamesrunning on game consoles or PCs.

FIG. 2 b is an illustrative correlation table of positions for twoalternate ways to play a certain classical video game while exercisingon an exercise machine.

FIG. 3 is an abbreviated chart showing the flow of activity in oneexample of a camera based computer system of the invention.

FIG. 4 a illustrates an interactive workout video session.

FIG. 4 b illustrates a weight lifting exercise embodiment.

FIG. 5 illustrates a simple device used by a senior citizen seated in achair, or persons in wheel chairs, which uses head or upper bodymotions.

FIG. 6 illustrates representative social embodiments of the invention.Further illustrated are methods and apparatus to control an avatar of auser or someone in a social networking site or other venue and wheredesired to incorporate this function into a game involving exercise.

FIG. 7 illustrates an aspect of the invention to help arthritissufferers and further to help them regain some degree of their formerabilities.

FIG. 8 illustrates an airplane game embodiment on a treadmill.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

FIG. 1 a illustrates an exercise bicycle embodiment the invention. Thishas been mentioned in other of my patent applications, and is herediscussed in some detail. There are basically two kinds of indoor bikesthat are presently used for exercise-a stationary exercise bike and asort of road bike, which allows you to simulate riding down the road. Inthese latter cases are also training devices that are used to allow youto bring your own road bicycle into your family room so to speak andwrite it there by having rollers allow the wheels to move without goinganywhere. This discussion is mainly going to focus on the use of aclassical stationary exercise bike but it is understood that theinvention applies to other bikes, and a large gamut of other exerciseequipment which the user moves against a resistance with either hishands or feet or both.

The invention may be used monitor the position of one or both pedals, amember moving with them, or the feet (or ankles or legs) of a user andthus by measuring the motion determine the rate at which the wheel isbeing turned by the user. Other exercise machine related variables likea degree of resistance (also called tension) dialed in can be monitoredif the movement such as a lever position to execute changes inresistance can be seen by a camera or other electro-optical sensor ofthe invention. Similarly other features may also be so monitored withthe sensor, such as optional features added such as game control leversor buttons. Location of any steer able handlebar present can also bedetermined. If desired the position of the rider in terms of seat orhandlebar height can also be measured. Person variables can also bemonitored such as head location and orientation, tilt (lean) of theperson to one side or other, hand position, arm state (straight, bent)neck location, waist location feet location, knee location and othervariables if desired.

As shown in FIG. 1 a is a top view of a stationary exercise bike 5, suchas a Schwinn spinning type having handlebars 6 and pedals 9 and 10 usedto drive wheel 12, via a chain or belt drive. A brake lever 13 and aoptional added game lever or button 14 are also shown. Movable portionsof the bike such as pedals, levers and in some cases handle bars may beprovided, In a preferred embodiment location and/or movement and in somecases orientation of contrasting portions are sensed using at least onecamera such as 30. Such contrasting portions may be for exampleretroreflector targets such as target 15 on wheel 12, target 16 on brake13, target 17 on lever 21, and target 18 on a turn able knob 19 that isused by the rider to control resistance against the wheel, and thuseffort to turn the wheel.

A video display 24 typically a large screen LCD or plasma display orprojection display is located ahead of the person, which display istypically 60 inches diagonal with a horizontal extension of about 52inches. It can be a very large screen such as 120 inch diagonalprojected on a wall. Generally speaking, I have found that more theimages on the screen approach life-size, in relation at least to theplayer, the more interesting the game is. As shown previously it ispossible to turn a HDTV 16:9 format TV to have its long axis vertical,in order that a person seen on the TV are more life size appearing inrelation to the rider and bike. This is very realistic for socialactivities and a more realistic feeling to the social interaction ismade possible as is described below. For example a 53 inch high screenof the above 61 inch diagonal example is about the size desirable forbike rider to bike rider interaction such as described in FIG. 6. A PC25 controls the display 24, as well as two speakers 26 and 27, and amicrophone 28.

PC 25 can be used as well to process images of camera 30 using machinevision algorithms known in the art (for example the MIL software libraryof Matrox corporation in Montreal, Quebec) in order to identify anddetermine where points such as those on the machine or the personexercising are located in the image field. Using this data, andpotentially data from sequential image frames, the computer 25 candetermine one or more variables relating to a particular point inquestion, such as its position, position relative to another point,velocity, acceleration and direction of motion, Alternatively a secondcomputer other than PC 25 can be used for this image processing andvariable determining purpose and in turn interfaced to PC 25. Thiscomputer could in another example be resident in the housing of PC 25.

In one embodiment of this application, a single black and white TVcamera 30, having a Field of View (FOV) sufficient to see all points onthe person and bike required, such as a one megapixel PL A? 41 Pixellink company model is interfaced to PC 25 via fire wire, USB or othermeans, and is used to monitor points on the person and points on thebike in order to provide information to the game software, whose imagesare provided on the display screen 24 via the computer and sounds viaspeakers 26 and 27. Other black and white cameras or color cameras canbe alternatively used even those with only 640×480 pixels. (Which alsomay achieve higher camera frame rates as a result), This camera 30 isgenerally equipped with an infrared band pass filter 31 passing the samewavelength as the source (e.g. 820 nm) so as to reject light outside ofas narrow a band as possible around this wavelength and thus increasesignal to noise. Since good interference filters of this type areexpensive, it is often possible to use a simple black appearing plasticthat effectively cuts off substantially all visible light, but typicallynot wavelengths longer than the led. This generally serves to eliminatethe effect of florescent lighting or LED lighting, but does not rejectall incandescent for example.

In this particular version two near Infrared OSRAM LEOs serve as lightsource 32 mounted next to the camera which light source in thisparticular application is been chosen to have aIR radiation at 820 nmand is also controlled by PC 25. IR is used so that the light from theLED is not obtrusive to the user, as it is pointing as shown in thedirection of the user's eyes. This goal can alternatively be achieved byproviding the user with narrow band (eg 50 nm) filter glasses that blockthe LED wavelength, if a visible led or other source is used. In thiscase a LED source of any particular color that can be so blocked can beused. And this in turn allows ordinary color webcams to be used ratherthan a black and white camera, which could potentially lower the cost.(at a price of typically lower sensitivity and lower resolution—ok forsome games).

The purpose of the LED attached to the camera is to illuminate points onthe person and the bike, and possibly other associated objects as well.These points are advantageously, but not necessarily comprised by retroreflective targets, though other features can be used if sufficientlyvisible and recognizable at the speed desired, and generally within thecost constraint of affordable equipment for the customer in question.

An optional color webcam 34 may also be connected to PC 25 for internetimage transmission. Room lights are typically used to illuminate objectsviewed by the webcam. The TV camera 34 may transmit an image of theperson on the bike to a remote location where there is another personexercising, a therapist, or someone else who may wish to interact orrecord the activity. In a health care related embodiment this webcam (oranother located elsewhere) can be turned on automatically in the game ifunusual behavior is spotted, by for example comparison of movements orother actions such as voice, with norms for that portion of a game. Thisability to provide information of medical value, while allowing apatient to play popular “normal” video games, is a big advantage inaddition to the motivation for exercise. It should be noted that evenwhen people are playing a Wii game for example, a separate webcam ofthis nature can be used to monitor and record or transmit their movementrelated information. At some point in the future, such data could becombined with the wii software itself to provide added game functions.

To avoid the bright spots from the retro reflective targets, if used (asis often desirable to speed up and render more robust the machine visionprocessing), webcam 34 is optionally fitted with a band blockinginterference filter known in the art 35 to block the wavelength ofradiation of I R LED 32 if such is not sufficiently blocked by the Bayeror other filter array employed in the webcam to separate colors.

FIG. 1 b illustrates a side view of the bicycle game of FIG. 1 a, inthis case including the person riding the bike. The person 1 seated onbike 5 is in this example also equipped with retro reflective targets; atarget wristband 36 and targeted hat (which can be a sweat band forexample) 38 which has a 3 or four target set 40 facing forward towardthe camera and light source 30.

Some useful targets and locations for bicycle related or other games oractivity are for example shown in the figures below The target 15 onspinning wheel 12 is monitored to determine how many times per minutefor example it is seen coming back to the same position by camera 30,thus indicative of the rotational rate of the wheel. One canalternatively use for example a target such as 33 on the person's shoes(eg on a toe as well or alternatively if desired to determine wheelrotation rate). Generally speaking it is desirable to use features thatcan be seen with the least FOV required of the camera, thus optimizingthe camera resolution. Thus looking at wheel rotation marker 15 near thetop of the wheel rotation is preferred to looking at the bottom, sincegenerally the hands raised vertically define the FOV in the uppervertical direction (but don't necessarily have to, there could be a gamethat restricted the complete field to points nearer the handlebars forexample. While large cameras are now available, a common 640×480 pixelVGA one is generally satisfactory and preferred as there is less data totransfer and process and one can run faster. As speeds of Buses andchips increase, so may this number. Or the speed of the processing canincrease to increase accuracy and reduce blur (see also FIG. 3).

The 3 dot target set is further shown in FIG. 1 c and such a set allowshead orientation and location in all 6 axes using the single camera 30.The ROI boxes 41 to 46 in FIG. 1 c provide one example of the regions ofinterest which may be processed by computer 25 in the image of camera30. Box 41 includes the region in which the head target set 40 isexpected as part of the exercise game(s) to be played. Box 42 includesthe region where the target 15 is expected to pass thru.

ROI Box 43 includes the region where the wrist target 36 on the person'sleft hand when outstretched to the side, while box 44 is the region whenoutstretched and raised. Similarly Box 45 includes the region where thewrist target on the persons right hand i when outstretched to the side,while box 46 is the region when raised. The three target set is not nearthe hand locations as shown, but if confusion results (e.g. in a gamewith lots of movement required) a simple processing step can distinguishthis set from the wrist target shape, which appears like a line in theimage. More regions or other regions of the person or bike can bechosen, but this illustrates a sufficient number to conduct a spinningclass activity or to play for example, a car chase game.

It should be further noted that a machine could have a simple retroreflective or other marker related to the amount of resistance one hasdialed in, such as a target 55 on knob 56 shown in FIG. 1 b. Note we canadd other mechanical control functions to the bike (or other machine)buttons, levers which push down against a spring like a gas pedal,handlebar grips which are spring loaded and rotate a target like amotorcycle throttle control, and so forth. In each case the camera isused to acquire the data from them to provide a solution at lowest costand complexity, though other transduction means could be used andinterfaced conventionally to the PC 25 if desired.

If white light is used for sensing purposes, all movable positioncontrols that need to be sensed on the bike might be orange or someother easy to recognize color against background, which ideally would beblack such that room light or projected light substantially reflects offthe targets only. The camera can see that as well without any sort ofadditional transducers and their cost, wires and complexity. This sameability to is easily see things also can relate to other types ofexercise machines, which also have adjustable resistances or inclines orother things that are changed to increase the amount of work that onemust do to of operate the machine.

Some components used in the above embodiments and their connections arenow described. In this figure it is assumed all software for bothsensing and display is located in the same PC 25. In some cases shown inFIG. 2, there may be a separate connection to a game console (such asXbox) where game graphics and audio are processed in which case the PC(or other suitable computer) is generally used for just the opticalsensing and other input processing. With suitable access to the computersoftware of the game console it is possible to have all inputs processedin the console itself, as the disclosed method of processing is simpleand does not consume much compute power of the game console (or PC, orany other suitable computer).

Inputs include data from one or more cameras used to sense positions ofpoints on the rider, bike or objects, or to obtain video to transmit toothers (or record for future use in analyzing performance for example,)microphones to pick up voice commands if used in the activity, andinstrumentation such as Heart rate monitors such as those by Polar Corpor equivalent used, interfaced to PC 25 of device and able to provideall data on the display as desired. A transmitter belt can be usedaround the chest to transmit the data directly to the pc device. Thisallows the maximum knowledge to be able to be imparted to the bike riderfor serious training and exercise. Monitor of heart rate can also beused to flag potential unsafe conditions and warn the rider by audio orvisual displayed data on display 24 or both. Comparisons of heart rateparameters to previous data can be provided on the display 24.

Outputs include besides the display and audio speakers, for bothcomputer prompts, and voice from internet connected persons and thelike. Other outputs could include vibrators such as piezoelectric typesto vibrate one or more portions of the exercise machine, such as thehandle bars or seat in response for example to game activity (likedriving on a washboard road). Such things require specially equippedexercise machines and wiring, which otherwise is not generally neededunless LED type targets are for example used.

As the person 1 pedals the bike and undertakes both the exercise and thefun movements of any game associated with the bike, the various cameramonitored points move in space, and the positions of the points atvarious time intervals are obtained by camera 30. These image points areprocessed by the computer 25 (including software either speciallywritten, or standard machine vision software such as the MatroxCorporation MIL image library Information is obtained that monitors theposition or movement of the person or bike features or other objectsused. This information can be used for teaching or diagnostics or tomore generally to drive the display and voice related portions of a gameprogram. These programs are resident on a game controller or pc, andmight optionally be downloaded from the Internet or used to interactwith other persons on the Internet.

The first game to be described herein is a bike tour through forexample, the streets of Paris. The speed of the bike and rider in thedisplayed image is in this case a function of the speed of the realexercise bike 5 being pedaled. To turn left, the rider leans left inthis example, and to turn right, leans right. The change in targetposition of a feature on the person such as their head, or neck or otherportion is sensed, and the rider displayed on the screen turnsaccordingly. Such sensing is aided greatly in both speed and resolutionin normal room conditions by the use of the retro reflective targetdatum's described. Alternative to sensing the person lean to effect aturn, one can sense a hand movement such as the conventional right orleft hand movement, or some other movement of the person can be sensed.Or as pointed out, we can equip the bike at extra cost with movablehandle bars which can be sensed. We can also sense a three or four pointtarget set on the person's headband or hat and easily obtain a movementof the head, as well as the viewing direction. This could alternativelybe used for the turning function, or in a more sophisticated program, toactually act as the persons view point—who might look to the side forexample, to see some danger approaching. This is particularly effectivewith large screens which give an immersive effect, especially if theytend to at least partially wrap around the sides of the person, an extracost feature and normally not available in ones home. This can allow badguys to approach more from the sides where you might have to turn andshoot for example.

Shown in dotted lines in FIG. 1 a are optionally movable handlebars 6(dotted lines) employed for giving the bike a steering function re agame displayed on the screen. This movement could be locked for playingother games such as strenuous spinning ones, where movement might not bedesired. As shown, the rotational position of the handlebars can bedetermined with a single camera 30 through trigonometry related to theangle alpha of the camera axis (see FIG. 1 b). As the left and righthandle tips 58 and 59 of the handlebars move in unison one toward theother away from the camera as the case may be, the position of thepoints in the field of view changes accordingly for example. Resolutionof handlebar movement improves when a substantial angle alpha (eg >20degrees, and often more) is used.

The actual graphics displayed can be generated using 30 graphical means,and represent virtual situations like today's video games. But they canalso be comprised of real footage of scenes as described in my copendingapplications and patents. These can be used to provide a succession ofvideo clips, whose play on the display is a function of the actions ofthe player. Touring through Paris is an example, where real shots ofstreet scenes can be used. If you virtually bike up the Champs Elyseeyou see to the left and right the real image. This can even be made moreexciting, by having a clip of a car sideswiping you, where you have toturn off the street to avoid the danger, and then go up another street.This makes travelogues much more interesting.

It is alternatively or additionally possible to locate the camera andlight source (where needed) in the rear such as 50 shown in dotted linesin FIG. 1 a. This arrangement has an advantage that the light source isnot pointing at the eyes of the player, allowing then visible lightrather than I R to be used. This allows conventional low cost webcams asinputs and further allows color differentiation of colored retroreflectors, or other colored features of the person or bike. However,strong light along the camera axis from the rear (such as useful withretro reflectors) can hit the display and reflect back at the rider orwashout the display. It can then be desirable to move the camera 50 offthe display to bike axis 23, for example to position ‘P’. Generally itcan be arranged that most points on the person and bike needed for agame are visible from this position.

Note too that many shoes today such as Nike brand have retro reflectivetargets on the heel (for running or biking safety at night) which can beused here as well. But in general, with the rear mounted camera, theremay be fewer points that can be seen, and it is sometimes difficult tofind locations in the room to mount the camera and light source. Thustargets may also be desired on the toe or sides of the shoe, in additionor instead of on the heel. While the camera can be connected to the PC25 via Bluetooth or other wireless means, battery power can be somewhatdifficult for the LEOs as they use considerable power, unless one admitsto recharging fairly often. Thus it may be desirable for wires to beused at least for the power, which may or may not be easilyaccomplished.

One may employ an alternative (or optional) side mounted camera alsoshown in dotted lines 51 in FIG. 1 a. This generally doesn't both therider unless very bright For some types of exercise this is a preferredlocation, for example if a particular type of walking or running is tobe analyzed such as might be used in a treadmill related applications.In this example the shoes can have targets on the sides, for example tosee the motions of the feet. This allows persons jogging on thetreadmill to compare strides for example over the internet.

Using body signals (such as leaning one way or the other—sensed in thiscase by head motion going left or right), or using ones hands to signalor by providing steerable handlebars whose position can alternatively besensed one can also provide a game where you are bicycling down a paththat led us a simulates the Tour de France or some path through yourtown or other things that the such as mentioned in previousapplications. If you are actually peddling the bike in these situationsit can be a stationary bike. Or it can be a road bike that you put onrollers such that the wheels do not contact the ground. In any case, themotion of the wheels can be monitored as shown above as can the positionof the handlebars including rotation thereof if permitted, which are nowdiscussed in further detail.

Let's move now to the case of a group activity such as a spinning class,also called indoor cycling, studio cycling, or other names. Spinningexercise classes usually consist of about 12 to 20 people. Classes rangefrom 50 to 60 minutes and are set to music. The instructor takes theclass on an imaginary ride that includes all types of terrain thatproduces a great spinning workout. If one is a type of person whodoesn't care for traditional exercise classes, this form of exercise canbe useful. One of the many benefits of spinning is you feel like you'rekeeping up with the class because everyone finishes at the same time. Aspecial spinning bicycle may have a weighted flywheel in the front andmakes you feel more like you're riding on a real bicycle. The spinningbike allows you to adjust the tension to keep pedaling within acomfortable range for your fitness level.

A new feature now added by the invention herein is to allow thisspinning activity to occur with a virtual instructor, and/or with aclass of one or more other persons which further may be remotely locatedbut linked by the internet or other means. In this particular example aperson can ride their bike in front of the camera and the screen.However on the screen, we can also produce the image of a spinningmaster or instructor who would then give commands and suggestions to theperson riding a bike. These commands and suggestions can come from apre-stored program, but can be in direct response to the determinedpositions and motions of the person and they portions of the bike thatare being monitored by the TV camera and computer. For example, theinstructor might say, “let's sprint”. A person would then lower theirhead and began peddling faster. The camera system can determine thatthey indeed are doing this and in addition to determine how fast theirpeddling from the movement of the pedals noted that for a given bike andpedal locations with respect to the diameter the wheel. The rate ofmotion can be determined, and if the resistance of the machine is known,the amount of energy being burned.

If the program determines that the person is not pedaling fast enough,an indication may be made either visually or vocally, such as theinstructor might do that is to say pedal faster. Could also say the ratefor example, you could say you are pedaling at 10 miles an hour try toget to 20. And so forth. The stored program could say raise hands, andthen monitor how high you raise them. And if you did it at all. Whilejust words can be used, one can also have a virtual instructorgraphically making gestures, as well as moving his lips to say thesethings. All of this can be done using motion capture 3-D video and videoclips of actual instructors can be called up from memory as needed andalternatively used This is especially workable if one keeps the variousinstructions to a limited dictionary of such things. When one is usingdigitally created voices as opposed to actual voices one can then put inthe names of the person themselves to personalize more easily. It couldsay say Susie, you're going too slow.

And you can have a real live spinning instructor too, communicating overthe net. He can see all his class on a composite screen with all theirimages and see their performance data coming from the camera systems ifhe so desires. This aids him in making suggestions to the class. Whilediscussed here with spinning it could be ballet or other types ofactivity done this way.

In the above application relative to stationary bike use for spinningthere is no particular need to monitor the handlebar locations at leaston a typical fixed handlebar exercise bike. However one may add anauxiliary handlebar or modify the bike itself in order to createhandlebar position related activities. Or one can use another bikehaving such handlebars. Target datum's if on the shoe (eg toe, heelsides) can easily be monitored with the camera of the invention toobtain rate of movement, if monitoring pedals or other targets providingsuch data like is not desired.

Besides monitoring wheel speed and steering/handlebar location, one mayalso use the camera to determine real (that is actually actuating thebike brake such that one has to expend exercise to accelerate it again)or simulated brake position that can be used in games to brake the bike.For example the person whose character you are in the game may need tostop its motion, either on a bike, in a plane or car, or walking. Inthis case you need to apply the brake, though it might be of time valueto make it not be necessary to fully stop the real bike wheel in orderto effectively cause a stop of the character's movement in the game.Other game related modifications may also be done by application of thebike's brake or other bike functions if any. The same holds true forapplication of the invention to treadmills, Nordic tracks and otherexercise machines.

The camera based computer program resident in computer 25 can also beused to help determine that you should not go beyond a certain limit.This is especially important for rehabilitation and for seniorsexercise. This can be a limit in terms of the extension of your arm orthe number of times you extend your arm a certain distance for example.Or it can be limits that have to do with ones accumulated energyexpenditure. Other exercise related variables can also be monitored.This can be particularly useful for those people who have had some sortsof problems that they are trying to rehabilitate, for example strokevictims. Older citizens can benefit too. It should be noted that theinvention as well, if it determines some sort of an unusual activitygoing on or one that could be dangerous could also automatically queryusing techniques known in the art to Internet sources for medicaladvice. This source could come from the manufacturer of the machine orelsewhere, and can be not only medically related, but related to otherissues such as those of the machine itself.

In some applications, one can monitor either points on the person aloneor the person and points on the bike (or other exercise equipment), oron the bike alone, in order to obtain the desired information to controlthe game. In some bike examples it is possible to provide games whichjust use the points on the person, since all points on the bike whichare moving are in turn person movement related. For example if youmonitor movement of your knee or toe up and down, it gives the samecadence result as monitoring the pedal movement up and down on the bike.Movable controls mounted to the bike or other exercise equipment mayalso be advantageously monitored directly with the camera system(switches, levers, handlebar movements, rotating throttles onmotorcycles, etc), and fixed points on the equipment may be used for acalibration.

FIG. 1 c illustrates for example further types of target members of theinvention which may be useful in the activities disclosed, such as bikerace games and the like. For example collar target 280 useful fordetermining from the camera image if a persons lean left or right, andstand up or lean over in the bike, when viewed from camera 30. Atargeted belt buckle is useful for stand up motions in this example.Another example are ankle band target 84 and 85, which in a bike gameare useful only in certain circumstances where such movement can becorrelated to a game command, though the angle like the toe could beused for determining rotation rate. Targets may be either bands, orcable like, made of retro reflective material such as scotch light 7615or another scotch light glass bead type, generally cloth backed. Any orall may be used in a particular game or activity. In addition there arepoint, or point grouping, targets which may be used, such as the threeor 4 point target sets which may be on head bands, hats, the back of ahand and the like. Suitable targets allow up to 6 axes of position andorientation of the object to which they are attached to be determined asdisclosed in my referenced applications and patents. The retro reflectornear IR (820 nm) signal to noise ratio of a typical target in a daylightbright room was measured at 300:1.

A target may be part of apparel such as shoes having retro reflectivematerial on the toe and heel regions. Targets around body parts such asarms, fingers, neck, wrists, ankles etc can be bead type, with a ropelike material covered with retro reflective tape, or can be plasticbands for example of larger width with retro reflective tape on them.Necklaces may also be of retro reflective material. In addition, targetsutilized may be colored in order to code which target is which, ifnecessary, which works as long as a color camera or other color sensingelectro optical sensor is used to determine their location in space. IR“color codes” can also be used, if different near it sensitivity filtersare used on the camera, like Bayer filters for visible color. Targetsused may alternatively or in addition be coded by shape. For exampletarget shape could be a diamond, or for another example a circle ofretro reflective material 87 on the toe of shoe 88 having anon-retroreflective black appearing line 89 within it. It should benoted that targets can be irregular and seen as blobs by the camera, andtracked in much the same way. Non-retro reflective portions can actuallycarry information such as the clothing or shoe manufacturers name orlogo.

It generally is not required to code the targets to identify them ifexercise is conducted on a machine or in other position controlledsituation with limited ranges of human movement in the game or activity.However, it should be noted that coded targets help to discriminateagainst glints from shiny objects such as door knobs and chrome chairportions which can cause confusing reflections that can look lightbright retro reflective targets. If all targets were circles for examplelike 960, it is unlikely anything else would look like them, and thusthis shape can be used as a differentiator. Subtraction of backgroundsignals can be used too as discussed below. While more expensive, 30camera can also be used such as that of Canesta Company, and in thiscase objects in the background can be discriminated out by range.

As noted elsewhere herein and in previous work, bright contrastingcolored targets can be used, without benefit of retro reflection as longas sufficient light is available. The beauty of the retro material isthat in relatively low room light best for viewing the display, theretro material gives strong signal to noise without being obtrusive. Andthe Led's and camera use little power, thus potentially allowing them tobe battery powered and wirelessly communicate camera and led controlinformation to and from the pc or other computer.

As I've pointed out in the referenced co-pending applications, thetarget points can be natural features of the object itself, which arerecognized by processing the image data. Or they may be specializedpoints, which can either be part of the object for example light coloreddots (or other portions) on a headband or can be attached to the objectfor example a plastic retro reflector. Retro reflective tape or clothmaterial can be sold or attached to objects very easily and is has theadvantage that it is flexible. However plastic corner cube retroreflectors and cat's eye types can also be used. As I've pointed outpreviously such retro reflective targets are very useful in that theyhave of very high contrast with respect to their surroundings whenilluminated by light that is more or less on axis with the camera as theparticular light shown is as the LED or LEOs mounted next to the camera.Note that flash sources as well as continuous sources can be used, andincluding ones synchronized with the camera shutter for maximumbackground noise rejection.

The preferred feature of a person or exercise object to identify andtrack with a camera (or other electro-optical sensor) in conjunctionwith such activity is a retro reflector or a light emitting diode insome cases, and if power is not an issue. Retro reflectors provide veryhigh contrast with illumination on or near the axis of the camera,generally enough to distinguish them from any other sources of light ina room on brightness basis alone. However, there are situations such asglints from door knobs and the like, where one might have a potentialproblem. This would be particularly true it in a freestanding situationsuch as FIG. 4 or 5 where you were trying to illuminate a large areathat someone was doing exercises or dances or other things in. In thislatter application, it is also possible to use targets can bedistinguished by other features than just contrast. For example by theircolor or by their shape or both as has been pointed out in previousapplications and above.

The rope or band type arm, leg, head, wrist, or ring type targets allhave a general shape which can be recognized as apart from other thingssuch as typical glints, or white diffuse items for example. Their imageappears considerable longer in one dimension than in the widthdimension. And to further code them, one can easily make them have blackbars running across their width for example.

It is also possible as previously pointed out to simply use color orshape or both as a means of sufficiently recognizing the targets withrespect to a background. In this case, no special light IR sources areutilized and no special on or nearly on the camera axis lighting isnecessarily required. The targets appear in the image taken by theWebcam 34 (and may be used with the webcam 34 instead of camera 30 forexample), but this appearance of bright spots in the image (particularlywith retro reflectors, if used) may be perfectly acceptable in somecases. The drawback of using such features that are not high contrastreflectance is that typically one may have to run a particular computerbased system slower in order to process the image more in order torecognize the targets in the total image field. In addition, there maybe more danger of confusing the target with some other feature such as asomething a person would wear.

The case of an exercise bike such as noted here though is one wherethings are relatively constrained in this regard as the room with thebike and clothing are under control of the user generally speaking. Sothat a bright yellow target for example is generally discerniblerelative to the bike and typically the targets are generally withinknown regions with respect to the bike and can be looked for by theprocessing system of the computer in those image areas to correspond forgames that are set up to recognize signals made by the person in certainregions of space. This is an important simplification, lowering the costof the camera computer portion, and increasing reliability of use.

Even more generally, there are certain situations here where you mightnot need any particular targets at all. In other words, you could usenatural features of the bike or person. If you raise your hand up forexample this may be quite discernible relative to the background of theroom. If you move your head down or up, the same may hold true. Or youmight simply use at a hat that had a different color which can besearched for by the image processing program, to help in rapidlyidentifying your head. Thus it in summation, the situation and the gameitself may determine just how much specialized target application isrequired. And in some cases most or all of the targets one might employmay not be needed. In the same vein, it may be that one or more highlyvisible and contrasting retro reflectors are required if fast motions ofcertain points under any circumstance are to be encountered. It is alsoclear that as processing speeds continue to increase in the inexpensivePCs or game controllers one might use, then less and less specializedtarget information may be required in the future to operate the“exergaming” system of the invention at the speeds and accuracy requiredfor good motivation and enjoyment.

Generally speaking though, a goal of the invention is to providereliable performance at the speeds needed, from low cost equipment. Theretro reflector approach does this without wires or batteries as LEOsrequire (which also can give good signals, and give color coded ones,without worry as to bright light in the persons eyes. While retroreflectors provide high contrast when using coaxial incidentillumination, self-luminous targets such as LEOs can be used, which havehigh contrast in many cases too? LEOs can be battery powered say by abattery pack on the person and/or the machine. In this case they canoperate if desired in the visible range as there is no need for light infront of the user's eyes. The LEOS can be colored too, to allow theirseparation one from the others. And they can be pulsed at a frequency,which can be synchronized with camera shutter to maximize signal tonoise.

One can relatively easily also add highly visible features such as retroreflective targets or leds to accessories used by players of these gamessuch as retro reflectors on I POD ear phones for example, to be used todetermine head pointing direction or head height. Objects held by theplayers can also be targeted. It is for example possible to target aconventional game controller pad with a 3 or four point target set anddetermine its position and orientation in a game, perhaps used forflying a plane for example. Moving a game controller by smallorientations and positions, would defeat the exercise benefit of theinvention however, just as it can for the Wii example mentioned in theintroduction.

One or more fixed portions of the bike (or other equipment) can beprovided with fixed targets such as with target 53 and 54 shown in FIG.1 a on the bike frame. Such fixed targets can be used for set up,calibration or reference purposes for example. And these targets whenused with targets for example on the handle bar ends 59 and 58 whenhandle bar is fixed, or the bike is in the straight ahead position, togive a full 6 degree of calibration of the camera to the bike, as 4points of known spacing (e.g. the distance D) in a pattern which can beused with the stereo photogrammetry method of Pinckney et al and others.This means the bike or camera may be moved and the whole systemrecalibrated in the computer such that positions on the bike or ofpersons on the bike can be known with reasonable accuracy with respectto the image on the camera matrix array which is digitized and used todetermine these positions, and any changes therein which are desired,then in real time. If the handlebars are adjusted up or down, thisamount can be changed by entering the amount moved or using the cameraonce calibrated to the initial setting to measure the value of the newhandlebar position.

FIG. 1 d illustrates an alternative arbitrarily located camera 60 in thevertical (and/or horizontal for that matter) direction, which iscalibrated to suit either the geometry of the set up, or the gamefunction or both. This can be used in other exercise machines too liketreadmills, Nordic tracks, Stairmasters, rowing machines, and the like.In this example the camera 60 is shown at an arbitrary location at anapproximately 35 degree angle to the horizontal which also allows us todetermine certain motions such as a punching motion of the players arm34, typically in a direction more or less parallel to the bike axis andperpendicular to the display. The inset 62 shows the image on the matrixarray of the camera from a wrist target in three different positions andat different times: Right hand high overhead C′, punching forward B′,and on the bike handle bar A′. If these are the only three positions theright hand can be in for valid activity, in the particular game, then itis only necessary to see if the target is in one of those positions. Themeasurement can be with respect to the camera optical axis 57 or to alocal reference on the machine or other fixed object such as datum 53.

Bike and treadmill based games are generally good choices for exercise,as they leave the hands relatively free at least some of the time, suchthat one can use ones hands for various game commands. And ones head andupper body to a degree are generally available too.

If the person does not do a suitably extended arm punch, the image B′target point B will lie closer to the optical axis and therefore thecenterline of the camera array. And by the same token it will liefarther from the image 53′ of the fixed point 53, alternatively it canbe compared to its original point A at rest on the handlebar, orwherever the person was resting. Limits can be placed on just howdistant from the reference point chosen one has to be before registeringa signal such as a hit on a virtual bad guy on the screen or whatever.Similarly limits on other positions such as the hands raised position Cfor both left and right arms can also be set. The soft ware can operateby actually measuring the amount, or setting up an ROI where nothinghappens unless the feature in question enters the ROI and in this caseit trips a switch that that function has been done, in much the same asa game pad switch might do the same function in the game. As opposed totripping a switch, one can also activate the switch which keeps thefunction on as long as you are in that ROI location. For example, if oneuses the left arm straight out position to indicate the wheels of avirtual car on the screen should be turned left, then as long as yourleft arm was in the ROI for that function, it would cause the car tokeep turning left.

Another point is that you can have a special target, in which you have around target 64 on a mitten such as 65 which faces forward and isentirely different shape than the wrist band target (which itself couldbe on the bottom of the mitten, in one style). Then you look for thisshape target (and generally a bright indication there from as well) toindicate punching actions. Moving from a position on the handlebar to aposition out in front.

The use of a single camera with a simple 20 image plane to effectivelyobtain 30 data of this kind is made possible in consideration of aspecialized set of game positions possible, or through the use of targetsets or types whose baseline allows one to acquire a degree ofmulti-axis information from one camera using stereo photogrammetry. Inother words to effect a game of the type described herein on an exercisemachine, it is generally not necessary to see a whole range of possible30 movements anywhere in space in front of a camera, but rather justthose regions where certain types of movements or positions are likelyto occur as required by game play. This is an important simplification.

It should be noted that a camera mounted off to the side can alsoachieve similar results, such as camera and LED sources 37 mounted atangle gamma in FIG. 1 a with respect to the axis of the bike.

In relatively few occurrences or regions sophisticated orientationknowledge is needed. The ones most often of interest are first theorientation of the head (as opposed to just xy head position) whichorientation for example may indicate a gazing direction of the personand can be used to change the viewpoint of the displayed image. Alsouseful in many games is the orientation of a object such as a pistol oryour forefinger which one might point at something on the screen. Inthis case one can observe the shape for example of a cylindrical targeton ones fore finger or the gun barrel in order to approximately discernpointing direction for example. This has been discussed in myaforementioned cases incorporated by reference. When pointing with yourforefinger as a gun barrel, you can move your middle finger and cause atrigger signal to be detected by the camera, for example using movementof a ring shaped target retroreflector on the end of your middle finger.

Let us now illustrate more or less classical video game forms (availableon Playstation by Sony and XBOX by Microsoft) which can be played usingthe invention while exercising on a machine and thus motivating exerciseat very low cost compared to the cost of creating special exercisesoftware as might be desired for maximum benefit. It should be notedthat one can even use the camera and computer system of the invention toplay video games while not exercising, if so desired. This allows peoplewho are confined to wheel chairs for example to play, using movements oftheir heads, hands, neck or whatever can be targeted reasonably and seenby the camera. The invention also makes possible classical video gameswhen the person is not on a machine but rather in free space beingobserved by one or more cameras.

It is also noted that the person riding a bike (or otherwise engaging inactivity as disclosed herein), may as part of the activity holds someobject such as a pistol or wand or other game related thing (or exerciserelated such as a weight, which also may be used in a game) in theirhand, which they could raise up or move in some direction at would beinteresting for the a particular game. In this case the object can havehighly discernible targets, and perhaps it is not necessary to targetthe persons hand, finger or wrist though this depends on the game inquestion. Things that would normally be used in the process of theactivities would be the machine itself, an object a player would hold,and/or perhaps a piece of clothing for example. Other objects can forexample, be sporting equipment such as baseball mitts, tennis racquets,balls, hockey or lacrosse sticks, cricket bats, on so on. Or it shouldbe noted, they could be play objects attached to the machine for gamingpurposes. One example would be a plastic car steering wheel which couldbe attached to the fixed handlebars of a spinning bike (or the frame ofa treadmill as another example) and the steering wheel monitored for itsangular position by the camera.

Lets also consider that the person is playing a video game of theclassic first person shooter type which is running on an Xbox360 gameconsole, where the exercise bike riders avatar is displayed in the videogame image (typically at the bottom of the display). The goal is to movethis avatar (which may in the original video game, be a car) thruvarious streets or whatever, shooting bad guys along the way, and insome game specific manner amass the most points. The Xbox controls adisplay, and takes inputs from a PC which processes signals from acamera. The associated LEDs for retroreflective target illumination ifused, are not shown for clarity.

In a drawing similar to that of FIG. 1 d, consider that the player isnot wearing the targeted mittens of that figure, but rather holding apistol in his right hand and having a wrist band target on his leftwrist. While in this case the person's finger or hand may serve as agun, using say a cylindrical (or other multipoint) target as mentionedabove on a finger, to obtain effective 3D pointing direction, in thisexample, the person on the bike holds a model pistol with a cylindricaltarget on its barrel, which can be easily identified as such (since thepistol may be waved around and pointed at different locations on thescreen where enemies appear, as one example). This setup also makes useof a LED emitter target at a wavelength able to be discerned by thecamera, said emitter, with its battery to power it, located on the hatof the player.

In this game the hands raised position has for example, not beenrequired, and the lens (or camera chip, if digitally restricted) fieldof view is shown limited to a narrower angle than in the game situationof FIG. 1 d. This can provide improved resolution or speed for example.

Other game functions can also be sensed as well by the camera, forexample the position of an attachable play device such as throttle lever21 mounted to the handle bar in FIG. 1 a, or the position of, brakelever 13 as another example. A trigger actuated battery or magneto inthe pistol causes an IR LED inside the end of the barrel to momentarilyemit light, at a wavelength visible to the camera and thus able to besensed by it when the trigger is pulled. The light then appears in theimage array 62 as such shown as shown in FIG. 1 d. Other things in thissystem are the same as in FIG. 1 d. Just this function, the pistol, plusthe ability to track head position and orientation plus rotation rate,can make an exciting game. Note that some variables, for examplerotation rate, could be provided by conventional means if desired, forexample a shaft encoder in the rate case. This costs more, also tointerface to the PC 25, but it would allow the camera FOV to be reduced,if no other variable near the bottom of the drawing (e.g. toe kicks)were being observed in the game.

The target that the pistol is pointing at alternatively may be sensed byother means. For example a camera to observe a laser beam coming fromthe pistol and where it hits the screen, and thus the displayed objectat that position on the screen. This camera would be interfaced to PC 25which would then coordinate the information obtained with knowledge ofthe display graphics being provided to the display.

Attachable play control devices such as levers, switch buttons, sliders,knobs and the like, which may or may not have any actual exercisefunction, can serve as an alternative or additional input to the game inrespect to looking at a part of the person's body such as a hand signal,or to listen via microphone 28 and voice recognition software for averbal signal (such as bang, for a trigger function) to indicate thepresence or value of a variable in the game.

A person biking in the above example or a participant in any game of theinvention herein, can be sensed pointing at something on the screen, andthis can be used in game play too. For example, a Harry Potter scene inwhich his wand is pointed at a displayed person and a magical spell puton them, perhaps by also saying the word “shazam”, or somethingsuitable. The wand can have a cylindrical target or other artificialfeatures to aid its pointing direction too, like the pistol.

In an alternative game type, described here for illustration of the manypossibilities of the invention, one can provide a game where the bike isgenerally not steered as part of the game (though it could be). This istypical of what is called a rail shooter game. However, In this case thegame may if desired be played a bit differently. The bad guys appear onthe screen and appear to shoot directly at the bike rider in the room(rather than an avatar of same). The bike rider then shootsbacks—physically at them, much like in a real shooting gallery orshooting skeet for example. This assumes the presence of a large HDTVdisplay, preferably able to display the enemies' full size for maximumrealism. This can be achieved with projection devices today. And thelatest types of large plasma displays such as the 150 inch diagonal onedisplayed at CES in January 2008. The bad guys fall by the way side asthe bike is pedaled thru various hazards (the bike for example, could bedisplayed as a Humvee Jeep type vehicle in Iraq in the video gamedisplay).

Another input might be to sense if you stand up in the bike, thistriggers a signal for the Humvee to leap a chasm, for jump a buildinglike Mario brothers for example. If you lean forward in the bike, thisis sensed, and if so programmed the humvee lower itself and go under abridge, or alternatively this command could apply the brakes. Oralternatively this can be the case if you apply the bike brake, which issensed as noted.

The same camera system of FIG. 1 and other embodiments herein can beemployed for all kinds of different games and activities, and used inconjunction with many types of machines, or no machines at all, as willbe further explored herein. It should be noted that several machines canbe monitored if you have a few at home for example. Generally, If youkeep the field of view (FOV) of the camera the same (by using the samefocal length lens for example (you would simply move the machine youwant to workout and play on to be within of the field of view of thecamera. Or conversely, simply move the camera (and associated lightsources if any) to the machine. Or alternatively, have a camera andlight source on each machine and connect that to the PC or game console,or whatever computer is running the machine vision program to convertthe players positions and movements into game commands.

Classical Video games as just described are known to be very amusing tomany, indeed in some cases addictive. This is reinforced by social play,with others on a network such as the internet, where things are not onlycompetitive, but on a personal basis so. it is possible that theinvention can prove a huge benefit in causing people to far morefrequently use exercise equipment they otherwise would not use—and useit a lot.

The video game as played above can appeal to many persons, as there areso many games from which to choose. In an institutional setting forexample, (such as a mental health facility, where patients are bored,and taking anti-psychotic drugs causing weight gain) the exercisemachine or machines of the invention may be setup so that video gamescan only be played using the invention, and the exercise machine in itsfundamental movement mode. If certain criteria aren't met (E.g. bikewheel rotation at X revs/minute or more, or treadmill speed over anumber) the game won't work. This then causes the residents to want todo exercise, just to participate in the game. Even at low levels ofactivity this can make a huge difference. One can also make moresophisticated criteria too, such as 1 0 arm stretches a given distanceper period as part of the game. In an opposite vein it is also possibleto program the game to shut down, if limits of time on the machine ormovement or rate are being exceeded for example, which could indicate anunhealthy condition that could lead to heart failure for example. Theseprogrammed settings can be chosen for the individual player as well, whocan log in with their password for example, or some other means.

The discussion above indicates you can play video games of the classicaltype using the invention, using portions of your body or features of amachine or its motion which are sensed by a camera or other electrooptical sensor. FIG. 2 illustrates a method of utilizing thisinformation to run commercial video games running on game consoles (egXbox) or PCs. Many such games are controlled by use of a classicalsimple gamepad controller such as a Logitech precision game pad sellingfor about 10 dollars, but are here controlled by the invention toprovide a new dimension of exercise based gaming which aids weight loss,muscle tone and general physical and mental health, while also providingthe opportunity to diagnose illnesses.

Illustrated is a means to play an existing video game by using softwareto convert signals to those required to simulate the electronic responseof the game pad (also called a game controller) used with the gameconsole (Xbox, play station, etc) or PC. In this case we substituteoptically sensed human and machine inputs for the existing game padinputs. The aforementioned Logitech precision game pad has 4 triggerswitches, 4 thumb switches and a D pad, which is a specialized “D-Pad” 4way switch (see U.S. Pat. No. 4,687,200) Not all functions arenecessarily used in any one game. Some more sophisticated game pads haveone or more of these controls able to operate in an analog fashion, forexample changing resistance like a potentiometer in proportion to thedistance you push them in for example. This same function can be relatedin the invention to how high you move your hand for example, or how farto the left, or how much in angle you cock your head in any particulardirection, or any other suitable parameter which may be sensed. It isalso noted that some game pads have force feedback in the form ofvibration of the device. This can at some complexity be added to theexercise bike too, in order to shake the steering, or vibrate the seatfor example. And audio feedback can be used via speakers.

An emulator “black box” 200 can be used as shown in FIG. 2 a whereaccess is not available to the game code directly from the PC 210 (orother processor) processing the information from one or more cameras 220used to process the point locations and movements of the person and/ormachine as shown above. The emulator is programmed to correlate theinputs from the human and machine in a defined manner to the game inputsneeded by the game program, which was originally written forconventional game pads or other such apparatus. This correlation (seefor example table 2 b) can be chosen by the user or others in whatevermanner is desired to make the game interesting or useful for therapy,exercise, diagnostics or other purposes. In short, even though one mightarguably lean to the left, to go left, it might be programmed in thiscase to have you move your left arm a given distance sideways, in orderto exercise that arm for rehabilitation therapy purposes. For mentalexercises, it might be you would choose some other action, like movingyour right arm, or finger, or a lever. To illustrate this, an alternateset of movements to play the same video game pad analogy controls isalso shown in FIG. 2 b.

For example, FIG. 2 b includes a table of correspondence between astandard type video game pad and an optically sensed position relatingto a car chase game while played on an exercise bike. A left turn can beindicated by extending the left hand fully outward, or by leaning thehead left. A right turn can be indicated by extending the right handfully outward, or by leaning the head right. Car speed can be controlledby sensing the rotational rate of a point on the bike wheel, or bysensing the rotational rate of the toe of a person riding the bike. Carbrakes can be controlled by sensing the brake position on the bike, orby extending a hand upward to slow the virtual speed in the game, butnot the actual bike wheel speed. Rocket deployment can be controlled bypunching a hand straight forward. Note that in both Set 1 and Set 2 inFIG. 2 b, the head orientation can be sensed independently, allowing theview point on the screen to be changed accordingly in the program.

It should be noted that the choice of positions to trigger equivalentgame pad controls in the game, can also be chosen in consideration ofthe ease and/or cost of image processing. And as the invention becomesaccepted as part of game console play, the variety of motion signaturesthat can be used as inputs can be much higher than those one mightequivalently do on today's game pads. For example, one can program acertain input to the game to have occurred if the player makes a “Z”gesture with his hand in space in front of him while riding the bike ofFIG. 1. This requires higher tracking speeds and more sophistication interms of potential confusion of data from other portions of the body orbike. These issues at the present time add cost and complexity.

Another such gesture is a throwing motion, or a racquet swinging typemotion, common to sports games such as MADDEN NFL by Electronic Arts forexample. A person on a bike can make a football throwing motion, whichfor a realistic throwing experience can for example be determined fromsome starting and finishing point in space defined by the wrist of theplayer in making the throwing motion. This is better and more naturalthan possible with a game pad today. The resolution and tracking ishelped because the bike has fixed the player more or less in a givenregion of space. This is actually a big advantage of the invention inthat I have been able to localize the processing requirements allowingsimple reliable equipment and ease of setup.

With more advanced image processing than normally required to make gamepad type inputs using ones exercise, the ability to track sophisticatedmovement with the camera allows us to place values on the “quality” ofmovement, relating to path traveled and the timing of movement at pointsin the path. One can modify the game in fact to reflect this, with thequality of pictures and sound, or some other variable output dependingon quality of movement and/or location reached—the simplest case being abig rapid stretch where you bike to attain maximum speed. A complicatedsports movement would be another example. For example taking a tennisswing while biking. Good swing, gets score, or hit in the game. Or abigger score, or some other choice of reward. All aspects of theexercise can be scored as the basic data is being taken. If desired,heart rate can also be monitored and added to a mix of scoredeterminants.

A very interesting feature of the invention which is subtly illustratedin FIG. 2 b, is that the same game, lets say a game that a person reallylikes and regularly plays with friends, can be changed in its commandstructure to provide a different form or level or other aspect ofexercise. This can be on the same bike, using different hand motions oneday, and going fast the next with hands on the wheel actuating handlebarmounted levers. And then the game can be played again, but set up on acompletely different machine such as a treadmill—an exercise machineoften found in the same location as a bike. The self calibrating aspectof the invention makes this easy to move, or one can just have camerasand light sources at each position, and switch over the computer and/orgame console used. And even more static forms of exercise activity canalso be used as game inputting devices, such as elliptical machines,bowflexes, etc.

The player of games such as those above, could using the invention besomeone in a wheel chair for example, who can control the game asdisclosed above for example by using their upper body motion (e.g. byseeing a neck collar target such as 280 in FIG. 1 or head band or hattarget or their arms (which could have an upper arm band target such as539 in FIG. 5) or targets on their hands or fingers.

It is noted that on a bike or treadmill for examples, the rate of motionis proportional to the exercise and in general, the calorie burnundergone. The game itself can encourage higher rates at certain pointsin the game for example, the bad guys car might speed up, or they mightrun faster or whatever. To keep up, you would have to pedal harder. Ifyou shifted to a lower resistance (Eg using the resistance knob of thebike as disclosed) in order to do this, you might say lose points, orhave less bullets to fire at them or some other game variable would bechanged in relation to you changing an exercise variable.

This also holds true in social games such as spinning where you areplaying or otherwise engaging with other members of a class.

One can while biking or tread milling (or in general) use a free hand tooperate a hand puppet like FIG. 5 of referenced application 20060033713.This in turn can allow me to control a character in a virtual world suchas in a game. The moves I make in a natural way learned in childhood formany, may be reflected in the character on the display which could be myavatar or a game character of some kind. The rate movement of the bikefor example could be proportional to the rate of movement of thischaracter in the forward direction. The 30 rigid body position of thehand puppet can be sensed as well has its head and hand positions withrespect thereto as shown in the reference.

It should be noted that the game aspect of the invention can be furtherrelated to exercise goals by only allowing you to enter a command, ifthe movement required is correct. Or in another example, allowing it tobe entered, but giving it less of a value or weight in the game, a pointthat encourages you to do more movement the next time. For example (ofmany possible) a shot fired by the pistol could fall short of itstarget, if the bike was not being pedaled fast enough.

It is now of interest to consider natural moves one can make whileexercising on machines that might fit the classical video games createdheretofore for use with conventional switch and potentiometer type gamecontrollers. We have already illustrated biking faster to make a car,plane, or other object on the screen go faster. And applying the brakeon the bike, to make it go slower (and the bike too, but maybe in acontrolled and less abrupt manner). Another move is clearly to fire aweapon at some graphical image on the screen, while riding ones bike oron ones treadmill. This is another natural move. More natural moves arehand signals learned for car driving to turn a virtual car on thescreen, or to lean right or left to make any moving image on the screengo in that direction.

Moving ones head side to side or up and down is a natural way to turn,or rise or fall an image graphically displayed on the screen. This canbe used to race cars, leap buildings, and so forth, and can be done bypersons who are confined to chairs or beds.

Turning ones head to look one way or the other to cause a view pointchange of the display on the screen is natural too. Pointing ones feetin a direction of turn or up and down like the head motion is possibleas well, as it is for the hands. Both can be easily tracked in 6 axesfor this purpose with 3 or 4 point target datum sets thereon. The armcan be used for this purpose too, also with a target set on the hand, ora wrist band target.

Leaning ones whole upper body right or left may be used to cause in agame a car on the screen to turn, and one can twist ones upper body toaffect this as well. Indeed with a 3 or 4 point target set on oneschest, one can make, with some difficulty, all 6 degrees of freedom ofmotion commands with ones upper torso.

Set up of the system can be accomplished a number of ways, all madeeasier when an exercise machine is used as a base for the gaming andexercise activity since it fixes the location of the person within somelimits, and provides fixed points of reference for those limits withinthe camera field.

A first example of a setup procedure is now described relative to thebicycle exercise and associated gaming activity of FIG. 1. In Step 1,the camera and light source are positioned at an approximate distancefrom the bike, measured with a meter stick or other means, and aimedapproximately at the mid point of the expected target locations.

In step 2, the computer such as 25 reads the camera image of referencepoints (such as 53, 54, 58 and 59 whose dimensional pattern on the bikeis known a priori and stored in the computer) and calculates theposition and orientation of the bike to the camera using single cameraphotogrammetry such as described by Pinckney. At this point, thecomputer knows where to look in the camera image for the other bikepoint such as the wheel target 15 let us say from a previous step. Ifnot the computer using the loud speaker tells the user to put hold atarget first on the wheel, and after storing that image, tells insequence the user to try each other one, until all are known in relationto the fixed points on the bike. This step does not have to be repeatedas now all points on the bike are known In step 3, the rider sits on thebike and the computer in sequence asks him/her to make the motionsrequired for the game in question, or if a general setup, to go thru allmotions needed for the game library, so nothing needs to be re setup.The person then on prompting may raise his left hand up high over hishead, does so as a natural motion. The camera and computer registersthis location, and sets up a reasonable ROI such as 44 in FIG. 1 caround the zone where the wrist target looked at by the camera isexpected.

Upon completion, it then prompts the person to make the next move. Theprompt can be vocal, or it can be shown on the display, also graphicallyshowing the user what to do using an animation. Prompts until all movesneeded to operate the game have been so registered in computer 25. Atthis point the game is ready to run, and the limits of motion needed totrigger events can be set in if desired. For example, the user might saythat only at the very maximum arm extension would it trigger a gameaction, or another user might say if his hand got within 5 inches of themax extension, he would like to trigger. The computer such as 25 canthen use these limits in the game.

Similar instructions and prompts can be provided to explain to a personhow to do an exercise, and to check or correct them if desired.

A modification to step one, can be to have someone just position thecamera relative to the bike until the image of the reference points ison a certain location of the display on the screen of the camera image,corresponding again to the game in question. In other words the userwould be prompted to move the camera in angle until the camera opticalaxis 57 in FIG. 1 d more or less intersected the rider's chest. Thissetup procedure generally can work for any exercise machine, and is notlimited to bikes.

If the camera has a zoom capability which is not really needed with mostexercise machine examples, the zoom too can be calibrated in this wayusing the camera image to define how zoomed in the camera should begiven the extension of the persons arms needed for example.

An abbreviated method of calibration is to just let the machine set theROis using norms for people, without requiring step 3. This is fairlyaccurate especially if one enters ones height or another pertinentvariable (by computer keyboard, voice, or other means, even a handgesture). Unlike the step 3 procedure, this does not however account foruser adjustments of bike parameters such as seat height or handlebarheight which can otherwise be taken into account.

If by chance the person has moved into a position where the computer hasset a ROI for a neighboring position, a prompt can tell this to theperson—also visually on the display, and ask him to shift slightly inthe movement he makes. Or the ROIs can be reset to not overlap in thecomputer. This assumes that simplified ROI based processing is used, andnot something more sophisticated such as individual targetidentification and tracking.

It is noted that the invention comprehends designing a game such thatthe movements can be simple, for example to preclude overlap of ROIregions above when making the moves needed to make the game commandsneeded for the game in question. It is also noted that in time specialgames may be developed to take advantage of the capability of theinvention to make inputs to games in new and exciting ways.

One can actually use ones own feel and experience with a particularexercise to calibrate some aspect of the machine or the game program.For example if you want to make a jump out of the bike seat a input tocause a machine gun to fire in the game, in order to perfect yourjumping skills you can just change the program accordingly, overridingany preset program which might have the gun fired by a hand movement forexample.

At this point I would like to more generally discuss the imageprocessing and data gathering aspects of the invention. When clearlydiscernable targets are used, it is often not necessary for the camerato be well focused on them, though if small codes with in them are used,or targets are very close together, this becomes more important. Forsimilar reasons, the optics can be plastic and low cost, and do not needcolor correction if an IR or other single effective wavelength source isused in conjunction with the camera.

The system as a rule of thumb needs to respond to approximately 20-30times per second for each point, or in other words all the points haveto be acquired and processed to determine their location in 1/30 secondor so. This generally takes good signals visible well above thebackground noise level of the camera, both electronically and due toambient light conditions in the room. For example, a bike being pedaledat 10 miles per hour, has a revolution rate for the driven wheel of 20inch diameter 3 revs/sec approximately. To resolve this to 10 degreesrequires nearly 100 readings/sec. But since the rotation data can beaveraged to get speed over some longer time period, 30 readings persecond or less will suffice. And most people don't go this fast either.

Another variable to monitor in some of the game motions described aboveis the rapid extension of ones hand in a direction. Many games need thisdata for a quick motion or action such as firing a gun at a bad guy.Here again if the readings are every 1/30 sec for the ROI in question,and the resolution is only plus or minus the resolution of knowing thata target has entered the ROI zone which could be either a switchfunction, or an analog function, that is where in the ROI is it, andwith limits set up in the computer for defining the point at whichaction takes place. In this case one has to get a quick answer without alot of data processing and this implies good signal to noise, as well asreasonable resolution of the camera such that the plus/minus one pixelerror is not large. This latter is helped by looking at a relativelylarge target “blob”. For example a target which is 1 Omm in diameter fora field of view of 1 meter×1 meter occupies 1/100 the field in eachdirection. For a standard VGA camera with 640×480 pixels this is roughly25 total pixels which can be used to register the event, not countinglens and blooming issues which may expand the number of pixels actuallyseeing light from this target.

A related target image enlarging issue however is blurring. This dependson the speed and the lighting method as well as the camera shuttermethod. Blur makes the target seem larger in the direction of motion,which may or may not hurt its resolution depending on what the game ormotion is. Blur can be minimized and efficiency maximized by overdrivingthe led source with a higher current but for a momentary time afterwhich the camera is read out in a synchronized manner. As long as theled duty cycle is kept within allowable current heat related limits theled will not be destroyed. Up to 10 times less blur and 10× improvementin signal to noise can be obtained this way, which is preferred to useif possible. Diode Lasers rather than Leds can also be used, safetyaside, and can have very short high intensity pulses to effectivelystrobe motion and eliminate blur entirely.

There is also a potential problem with background illumination in theroom. If an IR source is used, with a low cost band pass filter on thecamera, then visible light from florescent or led lamps is virtuallyeliminated. However incandescent lights and sunlight can under certaincircumstances cause background issues. Some of these problems, ifpresent, can be eliminated by doing a background subtract at high speed,subtracting each frame from taken with the LED off from a previous framewith it on. This works well in many cases but reduces the number ofpoints per second which can be obtained. It should be noted that ascanning laser camera employing for example a mirrorcle mirror brandmems scan mirror and a low cost red laser can be used, with a high speedphotodiode detector and band pass filter. This has very high signal tonoise, and may be desirable in brightly lit rooms or outdoors. Howevermost exercise locations with displays that need to be viewed by theplayer by definition require somewhat subdued lighting.

FIG. 3 is a chart showing one example of the flow of activity in thecamera computer system.

For added information on use of single camera or two camera stereotechniques see for example, A paper by Dr. H. F. L. Pinkney entitledTheory and Development of an on line 30 Hz video photogrammetry systemfor real-time 3 dimensional control presented at the Symposium ofCommission V Photogrammetry for Industry, Stockholm, August 1978,together with many of the references referred to therein gives many ofthe underlying equations of solution of photogrammetry particularly witha single camera. Another reference relating to use of two or morecameras, is Development of Stereo Vision for Industrial Inspection, Dr.S. F. EI-Hakim, Proceedings of the Instrument Society of America (ISA)Symposium, Calgary Alta, Apr. 3-5, 1989. This paper too has severalreferences to the photogrammetry art).

A machine may be reconfigured to different exercises both mechanicallyin how the person may exercise against machine resistance, but also withrespect to the sensing and computational software used to determine thelocation or movement of points on the person or the machine, and thedisplay software which may be used to provide motivation and interest inthe exercise. Bowflex by Nautilus corp. is one of many types of suchmachines that use cables pulleys and weights or tensioned members toexert force, but applied in different directions depending on thedirection of movement of your hands or feet, and the setup of themachine.

The Bowflex or other reconfigurable machine has the ability to do manydifferent sorts of exercises with one's arms and legs and can beconfigured in different manners. Your arms can go to the sides againstresistance or in front or vertically. Reconfigurable machines aretypically used in ones home in a bedroom or family room, similar to anexercise bike or a treadmill.

If you set up a certain resistance you can program that into thecomputer and in the easiest case simply say the words for the voicerecognition program such as Dragon Dictate stored in the computer forexample. “Resistance five”. Then when you move your arms against thatresistance to a certain extent, but the camera can measure it canRelates based also on the number of repetitions. The to do which it cancompute the amount of energy you have expended and its any other. Theinvention also is helpful in displaying and/or verbally annunciating toyou via a computer way file called from memory for example that yourmotion was let's say not ideal in terms of angular orientation orposition as you work the exercise is a helpful function of the inventionin using such a machine. In a further embodiment of that idea, one candraw up on the screen for example; an idealized 30 graphic where yourmove should be and then you try to match your actual video acquired dataof yourself in the movement to the graphical form. In a game, one mightthem “lose points” if they don't follow a correct path, or end up in theright location, or complete the exercise on time for example. And bymonitoring extent of movement, repetitions, and directions and so on,one can calculate the energy expended in the session. One can evenmonitor to check the weight or other setting and automatically inputthat, similar to the barbell in FIG. 4.b below

As noted in previous disclosures, the invention can also be used forfree standing activity of a person, without additional use of anexercise machine or other equipment. It is for example possible to usethe same sensing equipment as long as the speeds of motion are withinits capability, and the positions of features on the person can beeffectively seen by the electro-optical sensor, or sensors, used.

FIG. 4 a illustrates an interactive session of a user of the inventionrelating to motivation for a given diet or exercise regimen, andemploying automatic computer inputs of the invention, also providingdynamically changing data during exercise, which further allows one tosee the effects on ones appearance of continuing a given regimen, orchanging plans.

The system of the invention can provide either locally via storagemedia, or remotely via internet downloads, what is commonly known as a“Workout Video” to the user, but in this case tailored to the usersdietary or exercise needs, and capable of receiving voice, position andmovement inputs from the user. where such data is provided dynamicallyin real time, the workout video may be individually tailored, and evenmoment to moment paced for the user—just as if an instructor waspresent, who might say “you're not moving your arms enough”, “go faster”go higher”, etc. If it goes too fast for comfort, the user can ask thatthe video slow down, and the video source respond accordingly so theusers don't get discouraged. Rate, extent, speed of the video and musicwith it, choice of music with it and many other factors can all becalled up on demand, and in response—automatically—to the users ownactions.

A large screen display makes it more realistic, and speakers providesounds which encourage activity—even computer generated sounds ofencouragement like “way to go”, or something taken as a direct inputfrom the camera system which could identify that the users movementswere good.

For example, consider in FIG. 4 a a different scenario, in which aperson such a person 450 in FIG. 4 b and not shown here for clarity isdoing a workout in front of display screen 404 which ideally is like afull length mirror, so to speak presenting images of others more or lesslife size. Input video digital images, such as that of an exerciseinstructor leading a workout are provided on screen 404 together withappropriate sound on speakers 406 from a remote source 410 connected bycomputer 420 via the internet. (Assuming a high enough speed datatransfer link, to allow a remote source to be changed subject to inputsgenerated locally). For example as user such as 450 lifts his arms inthe air, his hands are sensed in their peak height extension positions(for example) by TV camera 430 (and led light source, if used) connectedto computer 420, which sensing can be aided by retro reflective wristbands as described elsewhere herein. The positions in this case are justregistered in X and Y, in the object plane of the camera, with noattempt to determine range, z, from the camera to the persons hands(possible with the “30” stereo camera of previous inventions or theangled camera arrangement of FIG. 1 d.

The arm extension and other x or y position values determined and therate of change therein if desired, are reported from the computer 420 tothe remote program source 410 (which could be resident alternatively incomputer 420), which then can compare them to norms for that section ofthe work out video being presented—perhaps as well consideringpre-entered data into the computer and thence to the program, of theperson, such a age, weight, stature, starting condition, and medicalissues if needed or monitored such as heart rate.

The program can then dynamically be changed, to go faster or slower,change programming entirely or whatever as a result of the input dataand the dynamically changing data of the person, in this case his handposition and/or rate of change of position. And as mentioned, data canbe fed back to the person as to how he is doing, both by voicegeneration in the computer, or by visual display on the screen, or both.

In addition, a score can be generated, just as if it was a video game.For example, a computed score for having the most motion, the most rapidmoves, the most acceleration during the workout, etc. and this score canbe compared to previous scores of the user, or to scores of othersnearby or around the world using the same program for example. Such ascore “76” is shown generated by computer 420 and provided on the upperright hand corner, in this case, of the display 404. Scoring can bebased on an integral of moves, a mean, or any other tally desired. Somescores directly relate to energy expenditure and weight loss, whileothers could be just for “fun” so to speak.

In addition the data entered from your exercise can be used to actuallyplay some sort of standard video game such as described above in FIG. 2and elsewhere herein

The invention thus includes a method of providing dynamically changingdata during exercise, which allows one to see the effects on ones,appearance of continuing a given regimen, or changing plans. This datafor example can be tabular data of projected measurements (such asweight or girth) or of projected variables such as heart rate. The dataas noted in my previous or co pending applications can also be projectedimages of a person in the future. For example, if one set in a presumedworkout of the same type for 30 minutes each day 3 times per week, thecomputer could provide a projected weight or other body dimension in 6months time say. Or it could even show an animated model of you,shrinking in girth over time. Or with muscles bulging over time, orother related changes.

It is noted that a useful place to set up the invention is in a gym orfitness club, both to kill time, and to provide motivation forimprovement. For example it is customary for many to watch TV whileexercising on a treadmill for 20, 30 minutes or so. This time can bespent using the invention, which can present the images for example on ascreen in front of the treadmill which can even be touched by the userif desired (using for example the touch screen technology shown in my copending applications and previous patents), while walking. One can alsosense the user with cameras, similar to what has been disclosed herein

FIG. 4 b illustrates another free form exercise embodiment of theinvention, in this example comprised by weight lifting A weight lifter450 in front of display 400 can for example, lift a barbell, 460, withcamera 430 sensing his actions using computer processing not shown. Thecomputer 420 provides via loudspeakers 406 inspirational sound and imagefor the weight lifter, and, as he lifts the barbell, the TV camera andimage processing program resident in computer 420 can be used toidentify the amount of weight lifted (by the number of weights on thebarbell for example), and the height that it is lifted, and how often itis lifted. This data can be used to calculate the work performed, andother relevant training factors. And such data, plus a pre-enteredknowledge of the person (and monitored information such as heart rate),can be used to predict the eventual outcome of the activity, or topredict the eventual size of various muscles etc. These muscles can besimulated on a digital model of the weight lifter as described above. Ifthe activity is unsafe (too much weight, tilted bar, etc) the computercan sense this and advise accordingly as well. Too much weight after toolong a time of exercise would be another such example.

The bar tilt can be calculated easily using retro reflectors 470 and 471(as examples of target points)—alternatively the weights themselves canbe identified as features to be used in the calculation) and the angleof the bar calculated from the positions of the retro reflectors in theimage of camera 430. If the angle is deviating from horizontal by aamount of interest, a warning or other information can be provided suchas providing an magnified tilt image of the barbell on display screen400. It is noted that a bar bell or other freely held exercise objectcan be advantageously used, since the angle of tilt can be monitored toprovide added safety and weight training data associated therewith. Thisis true as well for objects filled with liquid which can slosh aroundand add additional variables to the weight or other distribution offorces of exercise.

Clearly, the TV camera can be used for digitization of the activity of alarge number of exercise activities for the benefit of the user. Thedisplay can include a digital model of a person or and the modificationthereof, or could be any other kind of motivational presentation forexample, such as a score, a comparison to previous data, a relativerelationship to performance of a friend with whom one is having a socialinteraction session or whatever. The camera computer combination can beused for other beneficial activity as well, such as assuring thatmovements are not beyond preset or calculated limits or time integralsrelated to endurance and the like.

To illustrate the exemplary value of the invention for just one of theof the many potential rehabilitation exercises that need to beundertaken at us consider the potential use for improving on theinvention of U.S. Pat. No. 4,199,137. This device is made for exercisingan ankle in 2° of freedom of rotation vetically and horizontally plus orminus certain degrees from a straight ahead position. This relates tothe problem of “teaching” of the ankle so to speak. The point here isthat by suitable provision of a target datum viewable by the camera ofthe invention onto the toe for example of the foot of the person, awhole new experience can be provided to the person as well as movementdata taken in multiple degrees of freedom for the use of their physicianor therapist as to the exercise that they have undertaken in trying tobring their foot performance back to normal. In some cases the targetdatum isn't even needed as the movement is slow and the shoe movementmay be seen at reasonable cost using the gray level image of naturalshoe features. The display can be used to show the person underrehabilitation important data what to do, how far to go and how manyrepetitions they've made and can give them encouragement.

If the person is in the mood for a game it can for example be somethingwhere they do their rehabilitating repetitions in following a particularvideo character on the screen. It is also possible for the person in hishome doing these foot exercises to communicate to another person havingthe same problem, who might be in another country for example. Thesepeople can find each other, be it via social networking sites, and canundertake little games such as those mentioned above, where one of thecharacters for example could be moved by person A and the other person Bcould use his foot to control a virtual gun to shoot it for example, allthe while keeping the movements of their feet within the norms of theexercise. This is given as to be illustrative of the potential and notnecessarily something that a doctor might recommend. But the point isthat to make games around things that are possible and inspirational topeople as well as to cause them to socially interact in so doing if theyso desire.

It should also be noted that system can be programmed so the patient cansee on the display when they are approaching a limit of their goals.While these limits can have hard stops as described in theaforementioned patent they can actually also be programmed in softwaretoday for example a yellow were warning stop and then a hard stop for abuzzer with sound or a light was goal for something to tell the personthat they've gone far enough. Probably more likely is that the personhasn't gone far enough because of the pain involved in teaching the footonce again, and in this case again the making some sort of reward systemor interest level and going the full distance that is needed, but to thedoctor's advice is desired to also noted that the data about what thepatient is doing can be transmitted instantaneously over the Internet toa person in a rehab facility so the person does not have to be present.Indeed for example someone can be doing the exercise in question on thebattlefield literally, where the doctor being in a VA hospitalstateside. While this is been shown relative to the ankle movementsdescribed in the aforementioned patent.

The same sort of logic can be used for all sorts of rehabilitationexercises and because of the ability of the camera and optical system tomeasure its full 6° of freedom motions of the locations, and to measuremore than one joint it is also possible to have relatively sophisticatedtypes of rehabilitation as long as the person can follow the graphicsand other information that might be generated on the screen.

It should also be noted that the patient can see when they areapproaching the limit of their goals. While these limits can have hardstops as described in the aforementioned patent they can actually alsobe programmed in software today for example a yellow warning stop andthen a hard stop for a buzzer with sound or a light for something totell the person that they've gone far enough. Probably more likely isthat the person hasn't gone far enough because of the pain involved inteaching the foot once again, and in this case again the making somesort of reward system or interest level and going the full distance thatis needed, but to the doctor's advice is desired to also noted that thedata about what the patient is doing can be transmitted instantaneouslyover the Internet to a person in a rehab facility so the person does nothave to be present. Indeed for example someone can be doing the exercisein question on the battlefield literally, where the doctor being in a VAhospital stateside.

The illustration of FIG. 4 is also relevant to Wave exercise machineswhich shake the body, and make use of reflex actions in the muscles. Onecan alternatively consider for example the platform of FIG. 4 b to be awave machine that the person is standing on, generally grasping a barlike used on a treadmill. The games with hands and kicks and headmotions can all be played with the invention using this machine as well

It should be noted that the example of sensing of position andorientation of (and even identification of the type of) the barbell isjust one example of exercise objects freely held and supported by aperson whose position, orientation, or shape can be determined as partof a exercise or rehabilitation application of the invention. Anotherexample is a squeezable foam ball having optically contrasting targetsevery inch around its surface (at least in a direction that would beseen by a camera), viewed by the camera as one undertakes sometherapeutic distortion of the object, and sensed determination ofdistortion by the camera of the invention can provide both an input to agame, and be detected for teaching or compliance purposes. Anotherexample is a stretchable rubber cord with contrasting ring shapedtargets, which can be pulled like taffy as part of a rehabilitationregimen. Some examples of other freely held objects potentially movedaround in space for exercise purposes are batons, pom-poms, smallweights, and skip ropes. The Wii remote itself is also an example, forthose wii games in which exercise is undertaken.

FIG. 5 illustrates a related form of embodiment including an exercise orother game for persons in chairs which also is relevant to simpledevices used in the rehabilitation of injuries. As shown a simpleexercise device 500 with movable resistant arm 501 is used by a seniorcitizen 505 seated in a chair 510. Because the person, or device or bothare sensed, one can change the position and arrangement to suit the userand the particular exercise desired. And if desired the person can makegestures with the hand wrist target 505 (as one example) independent ofthe exercise which can be also sensed by the camera 515 and computer 520of the invention, if desired in order for example to change a normal TVprogram shown on display of the invention 550 or control some otherfunction such as ringing for assistance of a caregiver.

As one example consider that the camera 515 (and light source generallyused for retro reflective illumination as described above) can determinethe location of and to track the direction of view of the player in thisor other embodiments by tracking the head orientation from the four orthree target set 516 on the hat 517 of the player seated in the chair.This allows one for example to change view point of the displayedinformation, or use the head to for example control traverse of a gun inthe game, or in a completely different context to point at an icon onthe display. And the camera can sense such movement in both pitch andyaw, and can determine roll as well, tilting ones head from side toside. All these angles can be used to for example control an airplane onthe screen for example, and can be played from a chair. The camera cansense the target datum 550 on movable resistance arm 501 of device 500under the action of the persons hand 555. The arm 501 in this case is inthe form of a lever coming up from device 500 from below. In thesimplest case, the lever arm is moved back and forth and its position,extent of movement and if desired, motion path is determined. As before,a little game can be made out of it, which gives the person somefeedback as to how far they pushed it how fast and how many times theypushed it. Words of encouragement and graphical images can be displayed,and anything else like can help make it more interesting. Indeed thepushing of the arm could be something that causes something in a game tobe actuated.

The device 500 may also be considered to be a reposition-able device forrehabilitation of injuries, for use by therapists or persons in theirhomes. The goal here is generally short-term repetition of a particularmovement. Where the, the extent of movement and the direction may bevery important variables, along with the number of repetitions. Thenproviding an easy way to readjust the device to a new position in orderto assist rehabilitation or other exercise of other portions of thebody, or the same portion in a different way. Key to this concept is theability of the camera system to see machine portions to assist their setup. Portions can be targeted for example to aid their position and/ororientation to be determined by the camera of the invention (or anothercamera or cameras used for set up) and the computer may aid theirrepositioning by providing graphical or verbal instructions on thescreen derived from camera based sensed position data. In addition afterthe machine is set up, the camera system can be used to recognize amountof weights or other settings and advise and record them and the machinepositions settings used. Thus the machine of this type can be set upanywhere it is needed, and in any way required to undertake theexercise. Devices may be attached to chairs beds etc, wherever thepatient is—also in a vehicle, a hospital bed etc.

It should be noted that as noted above, the warnings can be providedusing the camera to advise the person that they moved the lever arm toofar for example, or they are about to. Perhaps to the limit that theyshould these inputs can be put in by a physical rehab person or otherhealthcare worker rather than the person themselves for example, thelimits can be displayed on the screen big warning labels big a voice cancome on an image as telling them that it's, they should stop or backoff. The image can be of the healthcare worker himself or herself or afriend or family member.

The PC 520 can be used as is known in the art to control a tv display550 and provide sound to loudspeakers not shown, or to pick up voicecommands from person 505 as desired and practical.

One can also use the camera of the invention to determine a pointingdirection of ones finger as disclosed in the referenced patents and copending applications. This is useful for pointing at objects on thescreen for game or selection purposes for example. Or you can look atthe screen and use the orientation of your head determined by the camerafrom target set to position a pointer on a screen displayed object ifdesired. This, for example, may be used by bed ridden people or thoseconfined to a wheelchair or a hospital bed.

It should also be noted that a trained medical professional can set up aparticular form of rehab exercise this way. By using some repositionablemechanical device to particularly be set up to allow a patient who is ina particular situation to use the mechanical setup in a way that willhelp them perform a helpful exercise. Just as a device was located nextto a senior's chair. It could also be clamped to the edge of a hospitalbed for example, for persons who are in hospital. This case, the camera,and the screen can be located at the foot of the bed for example. Andclamped to it thanks to the new modern flat-panel displays which arelight, or a video projection device mounted in the wall can be used toproject on a simple pull up screen located for example at the foot ofthe bed. Such devices can be used with most embodiments herein ifdesired.

It should be noted in the latter regard that in a machine may beadjusted in position in different ways one can also use the camera ofthe system and the ability to display easy to see graphic images on thescreen to guide the user to set the machine up in the first place. Forexample you could show the user, a new arrangement of the machine on agraphic TV display. Then you instruct the user to move a certainphysical detail of the machine (ega arm) to a point in space relative toits center, that is determined from the camera. When the camera sees atthe point is approximately reached that space It can tell the person tostop and lock it down there. The computer can actually depict on the TVdisplay the positioning with respect to the center of the machine suchthat you can actually see when you're approaching the right point andnot just hear a voice or other indication saying to stop.

The ability to help guide set up also leads to the ability to build newforms of machines. Even simple machines the only been require motors. Inother words, one can actually set up a machine having let us say erectorset like arms and other details. In particular positions that I might begood for a certain niche form of exercise. All of this since all of theset up can be guided this should be easy to do even in 3-D space. Inaddition, and a key issue is that the persons use of this machine isalso monitored. So that if there is anything about the actual devicethat that might mean some sort of cautionary aspect with respect toextensions or particular banding or weights or any other aspect thesecan all be pointed out to the user on the screen and monitored in realtime for help. This aspect basically allows one with relatively limitedmechanical equipment to create whole new forms of exercise, where theresistance is formed either simply by hydraulic or mechanical springs orweights on pulleys, compressed air or other means. Because one canreposition the arms one can also fold the whole thing up and get it outof the way it is for example in your bedroom.

In another version you follow with your hand or foot the path displayedon the screen, such as a straight path or a curvilinear path, in orderto undertake a specific type of rehabilitation movement in this casewith the person's foot having target on the toe 570 which may or may nothave resistance associated with it. A 20 or 50 path can be monitoredand/or displayed with the invention. Also one can sense the extent andorientation of movement and even the path if needed or desired. This isunlike the Wii device of Nintendo, which can be faked easily, as it usesacceleration rather than actual movement for many of its functions.

The invention can allow a built-in training function to be provided towhere some of the feedback to one might get from a personal trainer in agym can be provided in the home through computer programs. This can alsobe used to train persons in performing rehabilitation exercises.

For Seniors and others who nostalgically remember old movies, thedisplayed information can be composed of video clips from same. These(or any other video clips) can be pieced together to form a simple gamefor example. If the player/viewer wants the couple shown to kiss, shecan move her head (as one example of a gesture or expression which canbe used to trigger an action of the computer system) toward the screen,which movement will call from ram memory an immediate clip of a couplekissing—which might even be taken from another old movie. This same sortof thing can be done with sports games, where your physical motionsignal causes some action to occur, such as a hockey forward taking ashot at goal. Voice signals might also be used.

FIG. 6 further illustrates representative exercise based interactivesocial embodiments of the invention. As has been pointed out above, theinvention can be used to improve social functions and stimuli andtherefore motivation to the user. This in turn. When connected toexercise motivates them to do more exercise, which has huge benefits tosociety as well to themselves. Let us now consider the spinning class ofFIG. 1 and to make this more fun and bring it up to date in terms oftoday's possible Internet interaction abilities, let's consider aparticular case, where there are two players, Molly 601 is on exercisebike 602 in front of the display 605 and IR sensing camera 606 (withincluded light source) and web cam 607. The computer speakers and otheraspects of the invention utilized are not shown for clarity in thisexample. Molly, using the Internet facilities such as Skype or othersuch webcam type services, has called her friend Judy, who is in anotherlocation, and invited her to partake in a spinning class with her. Judy621 is on exercise bike 622 in front of her display 625 and it sensingcamera 626 and web cam 627 and computer likewise not shown. In thisparticular example, Judy and Molly can each see the image of the otheron their respective display screen. (And in some cases, such as in a gymthey could be physically seated side by side, but in this case they areremote from each other). For example, Judy's image is shown on Molly'sscreen 605 in real time due to the normal use of a Webcam and Internetservice. Both Molly and Judy can ride their bikes according to theinvention and their data in so doing can be displayed to them, and ifdesired to the other(s), as well as the video image. To illustratefurther capability, In this particular case, however there is a thirdperson shown on both their screens, and that is an image of a spinninginstructor, be it real or virtual.

This instructor can be presented in three forms at least. The first is alive image of a real spinning instructor instructing both of them and itis realized that there could be more than two people to instruct—indeedthere could be a class of 20 or 30, or 3000, although there is only somuch feedback and instructor can provide to individuals as the classgets bigger and bigger. Nonetheless, the instructor could see let us saysee 20 different images of persons in an internet held class on abig-screen monitor at his facility and be looking at them just as hewould in a real gym Or he could be looking at the data coming from thempicked up by the cameras and see how it is between norms to having theimages preprocessed for example to present to the instructor. It isnoted that one could actually combine and have a live spinninginstructor whose video appears on everyone's screen, and then assistancecould be done by the computer analyzing the positions, heart rate, orany other data from the person and in feeding and informationindividually to them.

Another situation is to have a completely computerized instructor inwhich no human instructor exists at all at the moment of the game. Inthis case is image would be either pulled from video clips of him givingthe actual lesson or generated using animation from 3-D graphics, whichare more adjustable to react to particular moves that one of the playersmight make for example if Judy did something really unusual. It's alsonoted that graphics and other clips and other information or data neededcould be downloaded from the net in a huge array of possible variations.Indeed, one could choose one spinning lesson from may be a thousandspinning lessons provided on the net from different countries anddifferent languages in different modes and so on. This also could leadto new forms of exercise equipment which could be adjustable indifferent ways also to fit a variety of instructors, languages andsystems that could be done.

In any case returning to the social issue it is felt that by havingMolly able to engage Judy to spin or otherwise perform exercise, whetherthat it increases greatly the motivation of Molly to a can presumablyJuty as well to undertake the exercise, the overriding goal. It's muchmore fun to exercise with a friend, and it's much easier to do so fromones home than it is to go to a gym where you're not only have theeffort of doing that but you also have certain social situations thatperhaps you might want to avoid or the equipment is tied up or itssnowing outside, or whatever. One of the advantages again of this wholesystem is that if you locate the device in your family room or bedroomor other room as you normally would, you can use the TV and Webcam andcomputer all for other purposes when you're not engaged in exercise.Thus the incremental cost is small, above the base exercise machine ordevice itself.

Optionally the machines themselves may be able to be modified using thisinvention. For example, let's take again the case of a bike whichtypically has a brake on the front wheel engaged by pulling a lever. Theposition of the lever can be sensed by the camera and as one puts moreand more force into the pressing of the brake pads against the wheelusing the lever of the handlebar the camera computer system of theinvention can sense indirectly and crudely the amount of force and thusresistance being applied to the wheel. This information could actuallyform part of the game where for example a spinning instructor says go toforce five and you would can pull your own lever to that force, thedisplay of force being presented by computer on the screen in front ofyou. The possibilities are endless, because the screens can be large andable to have a lot of data displayed as well as provide large images ofyour friends who are spinning with you so that you can have a feelingthat you are indeed in a real fitness club, if such is desired. Thecomputer can also be used to track progress, and relative progress(which could involve more than two people).

While spinning is something typically done by a younger persons today,the idea of social exercising as just described is not necessarily solimited. Similar motivational approaches disclosed can be used withseniors for example 80 years old doing exercise with their friends.However, the machine itself, either its type or its design or its sizeor other factor, may be less strenuous, with less impact or the like.One could almost even be the same program just scaled back in terms ofthe settings for how fast, how far, how much resistance and so forth.Perhaps with older persons, you would have an older instructor, whomight tell particular stories the related to do them the music that oneplayed could be 1950s songs as opposed to todays. Ditto the backgroundimages to one might use And so on. This latter aspect has anotherinteresting possibility in the sense that one can use the availabilityof such a program as an outreach avenue from a retirement home forexample, that might wish to help people who are currently in their ownhomes, but eventually would move into the retirement home. In this casethe exercise activity could be at accompanied by other sorts of thingssuch as health assistance, meal management assistance, and the like.

It should be noted that as before, the camera/computer system of theinvention can be used to score the players. In other words, Judy, couldhave made more rapid moments higher movements, and so on in which caseyou Judys score might be construed to be higher than Mollies. And therecould be handicapped given for various reasons to the players as well tomodify their scores. Such a scoring ability can be done dynamicallyalmost so that you can see that you might need to catch up by cyclingfaster toward the end of the session much as you might in a normal arace for example.

The above may be envisioned as a forum for social interaction of aperson typically in their 20s or 30s who enter acts over a socialnetworking site for examples such as Facebook. In this scenario, theperson might look up 5 of their friends on face book and query if anyfriend wants to spin? Let us one or more of the five would then say yes,and the group thus formed would form the spinning class so to speak. Ifone other person was present in the group besides the initiator thatperson's video image could be displayed on the screen of the initiator.If more than one were present each person could for examples see theimages of all of the others. These images could be arranged on thescreen as the person might choose. Perhaps one image larger than theother perhaps one image Central and the others around it. And so on.However, because spinning engage as an instructor who guides the classis clear that one would generally wish to display the image of theinstructor as well. But this invention does contemplate that one couldhave a class or the instructor simply was a voice (and musicalaccompaniment generally) conveyed to all.

For example, if Mary as participant number three for example is notraising her hands high enough. This would become apparent to thecomputer program which could compare her hands against a norm. That isthe height of her hands above the bike that can be determined. If onehas a datum on the bike such as target 53. Or one can actually teachrelative position aspect as a first step in the in the game, where Maryand all the other participants would raise their hands over their headto the typical highest point in the camera would simply record for eachwhat was relative to the baseline of the handlebars or other chosenpoint such as 53 or 54. Thus taught, the system simply runs a comparisonagainst that maximum point to determine whether the persons raisingtheir hands high enough or not when the instructor queries.

A data table can be made up, which is taken with the camera at each ofthe three participants Molly, Judy and Mary computer in this case looksat the data of each compared in the position to be taught value desiredsuch as a maximum value and determining the rotational speed calculatedby comparing subsequent frames.

In this particular one snapshot of each of the participants is clear toMary's hand position is clearly less than her optimum high. If theinstructor has as its in this case, the case asked the adjustments toraise their hands high. It would be clear that Mary is the one of thethree participants that has not done so. Therefore the program seeingthis it difference can save Mary you're not even close or Mary you'vegot to do better or other such comments. These can be chosen randomlyfrom a library of such comments relative to hand locations and simplythe names interchanged. If it's for example Judy rather than Mary, whosehands are not high enough in the air. Note that this approach can beextended to other activities too, such as sports, ballet, dancing andthe like.

Let us now consider an almost completely opposite set of persons,namely, a senior citizen Mrs. Jones 80 years old, who wishes to interactwith one or more of her friends in a similar manner to do a simplearthritis exercise with her hands. Because she has trouble walking as doone or more of her friends, she might be seated in a chair and engagingin the activity as shown in FIG. 5. She might find her friends throughan interaction internet site perhaps run by the Arthritis Society or arehabilitation group. In any case, having found two of her friends toengage in the activity, they go ahead and do the exercise just like thespinners above. An instruction program may be provided to tell them howto perform the exercise by moving their fingers in a certain way. Thisprogram could be either provided by the Arthritis society or in a database from the Google Open social site as mentioned or could be on a DVDor any other medium appropriate to the task.

In order to facilitate operation of the invention, Mrs. Jones mightadvantageously employ a retro reflective ring on her finger as disclosedin my previous cases Or she might have a ring on her thumb, or atargeted thimble, which would facilitate sensing of relative position ofher finger to forefinger, to determine a pinch motion. Or she could haverings on two forefingers, one hand to the other.

Now let us consider an extension of this idea to actually aiding aperson such as Mrs Jones to regain a lost skill and thus gain muchenjoyment.

For example consider the piano 700 of FIG. 7, being played by a person705 with arthritis who cannot now bend her fingers enough to hit thekeys properly. Camera 710 and computer 711 of the invention determinesher finger position and the computer 720 calculates the location and ifneeded, the trajectory of the finger in order to determine which keysuch as 730 the person would press with their finger 731 on the piano—ifit could comfortably hit the key in the first place. It then causes thecomputer to call a way file and drive a loudspeaker 735 to produce thatnote. It is generally possible to see finger location relative to thekeyboard fast and accurately enough with no added target, but one can beemployed, such as target retro reflector 740 on key 730. Nail polishsuch as 745 or rings or other such contrasting objects can be used onones fingers as disclosed in many of my patents and applications.

The piano in this example is further equipped with an optional LCDdisplay 750, ideally running the length of the key board. This can beused to provide a graphic showing a person who does not have innatepiano skills, where to hit the key in order to play a note of a givensong. This aspect is not limited to persons who cannot hit the keys, andmay also as an instructional too. When a person's finger has hit acertain key, the graphic can then move on and show the next note,building up the song as it does. This music can also be recorded by thecomputer or transmitted as desired.

The LCD or other display 750 can also provide an important function,namely presenting the music to be played, and then making the pages be“turned” without straining the player, either by the person moving theirhead, perhaps using a head target 755 sensed by the camera or seendirectly by the camera without need for the target, especially since themotion is slow. Another aspect is that since the computer knows what keywas last hit, when this key is at the end of a page, it could beautomatically turned, assuming the music was synchronized with thekeystrokes. This same idea can be used for typing keystrokes too, forthose suffering from disabilities, possibly supplementing voice inputsof words.

FIG. 8 illustrates an airplane game embodiment played on a treadmill Amodern-day treadmill used for exercise in the home or a gym has a motorwhich drives a belt that you walk out the motor essentially moves theroad under you and you have to keep up with it. Naturally you can dialin whatever speed you wish the keeping up part allows the device tocalculate with accuracy how many miles you are walking, and imparts arhythm to the exercise.

The provision of a motor on a treadmill has a disadvantage namely thatit creates a safety hazard. This is why there are kill switches on mosttreadmills with big red push buttons or safety chains. This being thecase it is hard to make a game out of a treadmill other than a walkinggame where you might steer (using your hands in the air or optionalhandlebars for example) something through by a roadway or city such asI've disclosed in co-pending applications. It is not easy to use oneshands in a situation like this however, because you need to keep up withthe motor, and should for safety hold onto the bar. One of the bigarguments for a motor is to facilitate calculating your travel, sincespeed and incline if used are known by the machine. But a motor is notrequired and many passive treadmills have been built. This sort oftreadmill, like the exercise bike can provide a safe and fun platform onwhich to build games to inspire people to get their exercise. And theinvention can track the person's movements and calorie burn, withouthaving to have a constant speed (for a given setting) motor driven belt.This means the person can stop, start, and even move backwards and stillmake this calculation.

In this example, however at the goal is to simply have a walk, in somecases perhaps against minimal resistance. One can while walking do otherthings such as use ones hands to perform exercises, also as part of agame. If the treadmill is large enough one can be walking and even goside to side a bit. With sensors and no motor involved, one can evenskip or do dances or other things that could involve even goingbackwards for a few steps if the game called for it. In short, one canmake a game out of doing many sorts of movements, all the while walkingin one's family room or other location.

The invention can determine location of object features and can feedbackdata to a person relative to their actions. If they go backwards as wellas forwards this can be sensed in the z-axis (longitudinal treadmillaxis) using either the angled camera trigonometric approach of FIG. 1 d,a side mounted camera such as 51 in FIG. 1 a above, or stereophotogrammetry or other 30 camera methods from a front or rearviewpoint.

Consider FIG. 8 which illustrates a person 800 playing a video game suchas an airplane game, (eg flight simulator or top gun or the like) whilewalking on a treadmill 810 having belt 811. In this example, the speedof the person driving the treadmill by walking on it at different rates(determined for example by tracking targets on the toes of his shoes 805and/or 806, or alternatively, ankle band targets 807 and/or 808) chosenby the user or the program to control the speed of the virtual airplanedisplayed on the screen and the yaw of the plane is determined in thisexample by ones wrist positions up and down in the vertical plane, withthe fully extended arms representing the wings of a plane which is.Wrist targets 815 and 816 are used on each wrist of the player 800 toallow camera 820 processed by PC 821 to acquire the data as to where thewingtips 815 and 816 so to speak are. You could dive down and raise upin the pitch direction by bending over or raising up your body, and thismovement can for example be sensed by monitoring the position and/ororientation of hat such as 830 with a 3 target set facing the camera, oralternatively by determining a drop in position of the wrists in the yaxis. As a further alternative the y axis position of a collar target835 around the neck or other suitable target could be used. Or one canuse natural features of the person, with no artificial target at all incertain locations, or even in all locations desired, if the imageacquisition was reliable and fast enough in normal room lighting todetermine the degree of bending over, or the angulations of the armpositions to control the plane. The treadmill allows forward motion ofthe plane to be controlled by simply walking at different speeds.Resistance of the treadmill can be adjusted to suit. If an incline isused or in general, the belt 811 can be prevented by known means fromgoing backwards if desired. It should also be noted that with suitableinternet bandwidth, image processing of camera 820 could be doneremotely at the internet server 870 rather than locally on PC 821 forexample.

The invention can be used for the diagnosis and treatment of mental,behavioral and other disorders. In addition, the invention in a relatedmanner also contains novel methods helping the mentally ill and otherpersons improve their quality of life. In the area of mental disordersfor example, the invention can be used to provide a digital 30 model ofa partner on the life size projection (or other) TV screen of theinvention. This partner can be programmed to carry on a conversationwith the person (who is often lonely and needs someone to talk with ordance with, or do something else with), with the dialog done remotelyover the internet (for example by psychiatrists, physiologists or othermedical professionals, or from pre recorded video clips from DVD storagesay, perhaps in response to questions asked and recognized by voicerecognition software such as IBM Via Voice. Alternatively or inaddition, the body model of the user can also be displayed on thescreen, such that the dialog between ones self and others (which couldbe many) can be observed in the third person. The model on the screen,which could be of another person, or the user, or both, can for exampledance, in tune with music, and this can be a dancing partner for theuser. Dance therapy of this sort is often valuable for persons who areill and often alone.

To effect animated activity of this sort requires more powerfulcomputers since the whole model (or significant portions) needs to bechanged dynamically, which is computationally intensive where 30 modelsare concerned. A Pentium 4 with 3 GHZ together with a high quality ATIor Nvidia graphics card. However one can do this job with simple 30models. Tai Chi is also possible with the invention in this mode, anddoes not in general require the response speed that dancing does. Tomake the tai chi session meaningful, harmonious sounds and pictures. aredisplayed.

One can in FIG. 4 b consider that the weight lifter 450 without theweights is engaging in freestanding exercise game taking place in frontof a large display, in this case a dancing game for seniors or personswho might wish dance therapy. In this case the dance “partner” is asubstantially life size depiction on a display such as a verticallypositioned RPTV or plasma display This arrangement can be used for othergames, such as a quick draw game discussed in my co pendingapplications, and does not necessarily have to be exercise related.

The dancing person can for example, be dancing or talking with the modeldepicted on the screen. The model could be that of a family member, afriend, a doctor or whomever. Or with an animation of a famous moviestar who the person likes. This interaction could also be done in thethird person, with the model of the person also depicted on the screenand having a conversation or dance with the other model. In this case itis generally necessary to sense the position and orientation of theperson in order to manipulate the image of the model—or the person—onthe screen. It might however be therapeutic to have the image driven bya computer program, which would cause the person to then say words inresponse to what he saw his virtual self doing. For example, making asocial error. This can be part of the therapy itself, which can includeteaching functions to help the person learn to better participate insocial activities. Or in a pure dance context, how to do a particulardance step.

One can provide voice or responses via the computer or remotemanipulation of data and in turn store the response of the patient bothvoice and movement for later diagnosis. In undertaking interaction withthis system, the images and sounds can be specialized to help stimulatesone mental activity, particularly valuable for older persons inretirement homes. For example one shows an image on the screen. You canmove your hand or your head to either it look at that image or to tryand touch it or indicated by pointing or other methods. These imagescould pop up at different points on the screen and you go ahead and dothat. Certainly you have to bring your brain into view at the responseof your brain is indicated by movements of your with body parts that aresensed by the camera again words of encouragement can be provided or youcould move onto the stage to or something like that. Some of thesethings could get involved were willing objects going across the screenwere moving and edit when they were doing certain things in theirmoments. Then you had to this. This sort of activity could be used formultiple purposes. One is to train someone a second purpose is to helpthem use their brain facilities, particularly an issue for olderpatients and a third would be for general amusement.

One can also do this in conjunction with other people. In this case, youare essentially doing that particular activity with respect to thesomeone else doing the same activity that someone else given theinvention can either be sitting next to you can be in another room ofthe same facility or could be across the world. Importantly, the personcould be a family member, and in the video context shown in FIG. 6 ofhaving video of the person as well as playing a game. This would be truehere to and perhaps even more useful for keeping in touch with yourfamily while doing something that might be in one way, family fun and inanother way, a helpful exercise for one's brain.

A useful aspect of the invention is that it can instruct people in howto relate to social situations. This is particularly a problem for thementally ill, who are often an extreme disadvantage and in the endbecome paranoid of social interaction. The invention can help them intheir home to learn the skills needed to go outside the home. Just asnoted above, one can use the TV camera to monitor the position of handsheads and other parts of the body, and we realize that in many casesthese positions and movements are actually part of the socialinteraction. We can use the camera to diagnose whether these movementsare correct in response to standard norms of images and voices that arepresented on the screen and thus determine with analysis if certainillnesses are present. But we can also use them to help guide some sortof rehabilitation training in this regard. This can also be in responseto those persons who have been injured in accidents and don't have fullpossession of either physical or mental faculties. The exact forms inwhich the video and audio information will be presented on the screenfrom the computer or from a live remote source or both depends on whattreatment is required. The key here is that the ability to have an inputwith both the voice and the physical positions of the patient isdesirable and available using the invention at a price that can beafforded in each home, where it really a lot of the treatment has tooccur both for cost reasons and for personal safety and embarrassmenttypes of reasons.

The use of the camera on the display or nearby allows you haveinteractive video sessions over the net, and to further incorporate suchimages in diagnosis—especially for remote diagnosis. The regular webcamlike camera functions can be turned on if desired when some sort ofnon-standard behavior (manifested as movements, positions or voice forexample) is detected.

I feel the invention may even be able to perceive subtle differences inmannerisms and expressions, with appropriate software to analyze motiongestures and further facial expressions as known in the art. Many thingsof interest are manifested as head positions hand positions etc. whichcan be elicited in response to stimuli on the display or audio ally inorder to identify some kinds of mental disorders at a far earlier statethan one might do it today. In particular those disorders that come outin a social way that is difficult to diagnose with children. If they arenot quite learning the social skills properly in the early years, it isvery difficult to tell normally speaking. But if there is a medicallylarge amounts of video positional and voice information in response toknown stimuli that the instant persons activity can be compared to, thenthere is a chance to do this and the payoff is to make a much betterlife for the person in the future via diagnosis and early treatment.

As one example of coding of targets to help in tracking feet with onecamera during complicated dance steps or other types of motions, one canhave a pair of shoes with retro reflectors on the toe and heel of each,which optimally can wrap around the toe and heel so they can be seen atany angle. The toe can have vertical stripes of no material while thefoot has black dots within the field for example. The right and leftfoot can be coded with respect to each other as well, if required.

It is also possible to provide targets or other features seen by thecamera system of FIG. 4 that can see on object underneath the feet ofthe person such as a rigid or semi rigid platform 480 with targets482-485 allowing the position and orientation of the platform object tobe determined, and intern movements in it which could relate toparticular dance or exercise routines. This would not be used generallyspeaking by persons such as the weightlifter shown in the figure.

The example of FIG. 4 also serves to illustrate what could be a sportsgame, with figure such as either catching a ball thrown, or throwing aball. This virtual ball could be thrown at the TV set, sensed by thecamera and a 30 display at the remote location present this ball to theother player to catch. This is not like some of my earlier work whichshowed throwing real balls (or hockey pucks or whatever).

As disclosed in previous applications one may devise a game in which theperson on the screen may be playing directly with you without the use ofan avatar in the virtual world sense. An interesting game can be playedwith a baseball pitcher pitching a ball at you the catcher, holding amitt that is targeted (or alternatively you could be a batter with abat). The goal is to catch (or hit the ball). The mitt or bat can beseen by the camera in up to 6 degrees of freedom, and the result of yourmovement in response to the display presented. Either visually oraudio—Like “yer out”!. A DLP rear projector may be used which canpresent the image in realistic 30 for example. The screen is mostrealistic if it is life size in the vertical direction to give it themost realism is a picture of a picture and a dynamic video the pitcherthrowing a pitch which I the catcher and to catch this can make a wholegame for the child that causes him to move around at least within alimited space such that it could be done in a bedroom or living room orsomewhere. Similarly the player could be a hockey goalie, with atargeted stick to assist the camera if needed. And the experiencedoesn't have to be sporting goods equipment or sport related One couldpretend to be a wizard in the Harry Potter movie, waving a magic wandaround, said wand tracked by the camera and the data there from used Inorder to perform tricks on the screen.

The invention with its sensing and feedback of information providesnumerous advantages, among them an ability to determine a usersperformance and via a computer program make suggestions or track theperformance or see if it is within bounds, or suggest deviations to makeit better. It can also send your performance information to someone elseover the internet or other means in order to compare your data totheirs, to drive an avatar or other representation of you, or todiagnosis remotely your actions. By combining sensed data, with actualwebcam gathered video images a complete experience and informationalbase can be developed.

While one can use a stereo pair of cameras or other means to get 3Dinformation as disclosed in my previous applications, in most cases weare trying to keep the cost low and the system simple and robust. Asimple single camera generally will do the job which is sufficient formany games Even where this single camera is a new 30 time of flighttype, like those of Canesta company or 30V systems company, the use ofspecialized retro reflective target points is advantageous to helprapidly located the points to be measured or tracked. Here too other ofmy previous inventions are helpful in that you can use the artificialtarget to locate one or more regions to be tracked and then use naturalobject features to determine other points of interest within a region.

Another point to stress concerning the invention is the fact of theperformance of multiple functions. This allows it to be shared amongst alarge number of different users and different uses for the same user andwith a commonality of function, the familiarity with it's use, and soforth.

From a cost point of view, it is clear that the same computer, display,and camera system can be used for a variety of applications, thus makingthe incremental cost of the invention herein relatively in expensive.Indeed the screen (the most expensive item, typically) can be used fornormal TV show watching or internet activity as well, or for control ofthe home Many people already own exercise equipment they don't usebecause its is too boring.

As stated above, an important point of this invention is to stimulatepeople to do the exercise itself. When you tour people's homes orfitness facilities in rest homes or mental health facilities or otherplaces you see bikes, treadmills and other exercise equipment but almostalways unused. It's clear that most exercise equipment is only used bytruly dedicated people. And that's the problem how to get more people touse the equipment? In order for this to happen the equipment has to beeasy to use and it has to be fun. It has to give good feedback, and ithas to be at the right price to where you can have it in the locationswhere people will naturally tend to prefer to use it. This again isanother focus of this invention because the goal here is to keep theprice low so that people can use it in the home and not just in a gym orinstitution. Indeed some of the features of the invention allow abuilt-in training function to be provided to where some of the feedbackto one might get from a personal trainer in a gym can be provided in thehome through computer programs.

Method and apparatus have been disclosed in co-pending referencedapplications. for improving the likelihood of success of a given healthplan by prediction of future appearance and providing an ability tomonitor the movement and shape of the user in order to tailor the plan,and the video or other instructions and stimuli given the user to suitthe situation. In any of the games above one may make a part of theplay, the projection of your appearance in the future as disclosed. Forexample, a person playing the game could see themselves in 6 monthstime, if the kept on as they are doing. If the person goes faster, orworks harder, they expend more calories also in certain places dependingon the exercise type, and variables such as these are inputted to themachine. It can be competitive as players can see how much they can lookbetter. Also other predicted effects like overall weight and endurancecan be displayed, not just images

It is noted that a sensing laser scanning projector or laser scanningdetector by itself as disclosed in my cases filed in august andSeptember 2007 is able to alternatively sense the points needed for thisinvention, and has and advantage in ambient light rejection, if this isa problem. Such a device, in projector form, can be operated from therear position 50 and used as well to project on a screen if desired.

The invention contemplates optional display of the digital model or areal image of more than one person side by side for example:

You and a friend.

You and a plurality of friends, the others remotely located with datafiles transferred over the internet or otherwise.

You and yourself from a stored image from an earlier time.

As noted in my August 2007 filed case, I mentioned that one caninterface a monitoring device to the person and interface this to thecomputer 25. for example, pulse rate monitor transmitter 35 or the pulsemonitor of the handgrip of the bike (not shown). Wrist bands or otherbody hugging devices used as targets can also serve as pulse monitoringpickups for such purposes, just as other body clothing can also serve astargets. The same can be said for bud type or other earphones from anIPOD which can have retro reflective tape or corner cube retroreflectors on them such that each ear can be seen and head location andangle in the horizontal plane determined by the camera and computersystem.

It is also noted in the regard of monitoring a players positions andexercise via the camera computer system of the invention, that criteriacan be set up so that a player doesn't don't get certain powers or otherbenefits in a particular game unless their motion or extension of theirbody or other aspect of the exercise meets certain goals, such as speed,extent, etc. rotation Another factor could be the integral of theirmotion over distances or other factors proportional to Energy expended,which could if desired be then translated into “energy” of their gameobject in the game, and so forth. Such criteria are a powerfulmotivation to keep exercising, particularly in playing against others.Because of this, it is desired to monitor heart rate by the PC 25 viawhat ever practical means, and make sure healthy limits are notexceeded, providing warnings to the player if there is a danger thereof.

It should also be noted that the one can display on display 24 theresults in 3-D stereo for example using “Crystal eyes” or othersynchronized glasses and a DLP projector running 120 Hz to provide 60 Hzalternate views to ones eyes. This then provides even more interestingexercise potentials in that you can react in “Z” or range axis tostimuli coming from the screen in that axis or apparently in that axis.This reaction can be using limited Z axis capability as taught in FIG. 1c using single camera photogrammetry or using true 3D camera sensingsystems such as via binocular stereo photogrammetry or time of flightrelated systems such as that of Canesta corp.

The invention, by monitoring the motion of a person on bike or otherexercise machine or jumping around may use this information to calculatecaloric burn rate, which may be more involved than just monitoringresistance of the bike wheel for example, and its velocity (thoughlargely dependent thereon). Using the invention you can input your goalsvia the computer, or via voice recognition or other means. Also couldhandicap yourself relative to others and the display can be programmedto show the goals entered, relative data to last session or others. Orprojected value or images at some time in the future if one continues xtimes per week. One may also display instant rates; instant heartmonitor if pulse or other measurement interfaces The computer can beprogrammed to provide you an instruction if you aren't meeting goal instretch or rate or other exercise variable. Also can, in a game, playthose in your level or handicap yourself with respect to their levels.

Competition is a major motivator to keep exercising. And this is muchmore fun in a social setting against someone seated next to you (eg in agym) and is still neat over internet. If you're playing a game and youstart to slow down your rate of pedaling the bike for example, the gamecould be programmed for you to lose points as a result. If your energyexpended in the game (calculated from factors such as integratedmovement, resistance, pulse rate, etc) for example began to taper off,an energy bar depicted on your screen could start to go down. The energybar could change size or color or any other variable desired. And itcould be compared to your competition on a bar graph on the screen forexample. Energy related data can be calculated from machine variablesextensions of the persons body in the exercise or both or any otherfactors. The invention can be used with multiple machine types and gameslittle or no hardware cost, without wires running on the floor or otherhazards.

The camera of the invention can view retroreflective datums or otherfeatures on ones shoes and other parts of your body, both for positionand angle, 30+ times a second with good accuracy at low cost—assumingyou use special targets either built into the shoe toe and heel, orside, or to into clothing or attached to the person or clothing, forexample with Velcro. Dual core/quad core PC computers today can processthe camera information concerning artificial targets or in some casesnatural features on a person or machine at the same time they drive thegraphics. A viable dance experience and training can thus result, andone can dance with a virtual partner or teacher, using a big screen HDTVturned with long axis vertical so that the image appears life size ornearly so (for example approx 55 inches in vertical direction).

It is noted that the means of using portions of your body to enter datainto a video game can be combined with other means, includingconventional means if desired. However the exercise advantages lie ingetting your body involved in playing the game. Various movements suchas raising up in the seat of a bicycle, or pushing back at somethingcoming at you (especially exciting with a 30 display such as Texasinstruments DLP projection TV provides). The computer can allow you toenter your goals, verbally using voice recognition or with a keyboard orother means. You can also handicap yourself relative to others who mayhave disabilities or be less skillful And you can use the computer topredict an outcome of continued play and exercise. In addition you canmake in software criteria that your game action is only entered into thegame if it is correct, or of sufficient speed or magnitude. For exampleif you raised your hand high enough. Alternatively if you don't, theentry into the game could be decremented in order to inspire you to dobetter for example. As noted above computer analysis of the cameraimages can determine the “quality” of the movement, not just the extentor duration. For example one might define in a given exercise quality asa path taken and timing Another application is that the game can showyou on the screen what you are to do, and then you are given a certaintime period say to do it, or judged on the quality of doing it, or both.

The invention thus have in many cases intuitive physical hand body orleg responses to already existing video game graphics designed forconventional game controllers—a big cost consideration as the exercisemarket is at the moment much smaller than the video game market. Thecamera of the invention can be used as well to make a relatively crudeface scan of the player, allowing facial graphics to be entered into thegame.

It's noted that in Italy or hands gestures are widely used. One can makegames that utilize such hand or other body part gestures in the games,including social games where the hands gestures are part of the socialinteraction. Note that the exercise gamer could optionally use his orher hands to signal the TV and computer, in the same manner as a remotecontrol.

The invention can be used to check for anomalies in activity vs standardnorms, or versus previous play of the same person. This can be used insome cases to determine if the persons health has degraded, or they areoff their Medicine say (for example manic play of the game, oralternatively lethargic). It may be possible to identify signs ofoncoming (or recent) psychotic episodes to allow timely and mentalhealth intervention. Abnormal responses can also indicate that a patienthas taken too much or too little medicine. If the game can help monitorthis, it may be possible to actually reduce dosage, also because of thevalue of physical games such as the invention in countering depressionand mood disorders.

The speed, magnitude or other movement variable in the game computer canbe scaled for different persons. Seniors or injured people would not beexpected to move as fast or as far or perform the complicated movementsthat others might asked to in a game to get a good score for example.The speed or magnitude to create a response can be scaled in a computersuch as 25 for example. However, in the game itself, the response timeof a bad guy say to ones movement may be fixed, if the game software isalready written and not software changeable in the computer. It isexpected that in time, software code from game companies will allow forvariations in timing and other variables to suit exercise based games,and further will allow code to be enhanced where physical or cognitiveresponses are desired to stimuli provided by characters or events in thegame. However today, the games are not this way. Where timing is aproblem, one can use a predicted motion of the person doing exercise, totrigger a game trigger. For example, if one in an exercise game is toswivel a virtual gun by moving ones hand to full extension leftwards,this cannot be done as fast as someone can move a gun by pressing aswitch on a D pad. So to get the same speed, we can monitor position andvelocity of the persons hand, and determine what their eventual positionwill be, and use that data as input to the game. Or we might just chooseto use velocity itself. The faster you begin moving your hand the morethe gun swivels, or the faster it swivels. Or whatever else you program.In this manner one can approximate the response time of the existingvideo games while still undertaking movements providing worthwhileexercise. In some cases change in velocity, acceleration/de accelerationcould be a variable to use. As camera frame rates improve, also helpedby strong target signals, the ability to determine velocity and changetherein improves as well.

It should be further noted that another way to use conventional videogames designed for conventional game controllers, is to provide opticalinputs for some functions that are not timing dependent in the gameprogram, and use the game controller for the rest. This requirestypically the user to stay more still and get less exercise, if forexample the controller was attached to the bike of FIG. 1, and only lefthand movements were optically sensed say, leaving the right hand free tooperate some functions of the conventional game controller. It ispossible though to consider another version where very fast commandscould be given (eg a trigger shot) by voice or with a wireless switchheld in one hand, which can be operated even though the hand is movingduring exercise.

The invention's combination in many embodiments of determining locationor orientation of both portions of person s and objects allows bettercombination of many exercises and games. For certain niches may be theonly exercise device a person will use—a completely un-served markettoday and also perhaps those most in need. (Disabled persons, certainseniors, etc). The social aspect of the invention may aid this greatly.

A reason for purchase of the exercise system of the invention is majorimprovement in motivation and fun, with much more capability andinformation provided, without much added cost—and perhaps no added costin some situations where high cost components such as the display orcomputer are justified by other uses (eg watching TV or surfing in onesfamily room or bedroom). Indeed for high end models, one can possibleargue less cost if the exercise equipment would have had a fancy controlpanel not needed with the invention (which panel can be replaced by thecomputer and display for most functions).

While the invention has been described in connection with numerousembodiments, it is to be understood that the specific mechanisms andtechniques that have been described are merely illustrative of theprinciples of the invention, and numerous modifications may be made tothe methods and apparatus described without departing from the spiritand scope of the invention.

The invention claimed is:
 1. A computer based exercise method comprisingthe steps of: providing an object with which a person exercises, theobject resisting movement of the person exercising; providing a computercontrolled video display viewable by the person while manipulating saidobject; electro-optically determining data concerning points on saidperson; processing said data to determine a variable related to one ormore of said points; using said determined variable, controlling saidcomputer controlled video display; and providing a warning to saidperson, wherein the object is a barbell.
 2. The method according toclaim 1 wherein said processing determines the acceleration of at leastone point.
 3. A computer based exercise method comprising the steps of:providing an object with which a person exercises, the object resistingmovement of the person exercising; providing a computer controlled videodisplay viewable by the person while manipulating said object;electro-optically determining data concerning points on said person;processing said data to determine a variable related to one or more ofsaid points; and using said determined variable, controlling saidcomputer controlled video display, wherein the object is a stretchablecord.
 4. A computer based exercise method comprising the steps of:providing an object with which a person exercises, the object resistingmovement of the person exercising; providing a computer controlled videodisplay viewable by the person while manipulating said object;electro-optically determining data concerning points on said person;processing said data to determine a variable related to one or more ofsaid points; using said determined variable, controlling said computercontrolled video display; and determining the number of repetitions of agiven exercise performed.
 5. The method according to claim 4 furtherincluding processing an audio input from said person.
 6. The methodaccording to claim 4 further including determining the extent ofmovement during a given exercise.
 7. The method according to claim 4further including providing a score to said person related to theexercise performed.
 8. The method according to claim 4 further includingdisplaying a path for the person to follow in performing the exercise.9. The method according to claim 4 further including the step ofelectro-optically identifying said object.
 10. The method according toclaim 4 further including a game related to the performance of saidexercise.
 11. An exercise system comprising: a video display viewable bya person exercising with an object resisting movement by the personexercising; at least one electro-optical sensor viewing a plurality ofpoints on said person; and a computer to process information from saidsensor and determine a variable concerning said points, wherein saidcomputer is adapted to control said video display based on saiddetermined variable, wherein said object is a barbell.
 12. The exercisesystem of claim 11 wherein said exercise is part of a rehabilitationregimen for said person.
 13. The exercise system of claim 11 whereinsaid electro-optical sensor includes a video camera.
 14. The exercisesystem of claim 11 wherein said video display includes a substantiallyvertical long axis.
 15. The exercise system of claim 11 wherein saidvideo display provides information concerning said person's goals. 16.The exercise system of claim 11 wherein said video display providesinformation concerning said person's prior exercise performance.
 17. Theexercise system of claim 11 wherein said electro-optical sensor includesa 3-D camera.
 18. An exercise system comprising: a video displayviewable by a person exercising with an object resisting movement by theperson exercising; at least one electro-optical sensor viewing aplurality of points on said person; and a computer to processinformation from said sensor and determine a variable concerning saidpoints, wherein said computer is adapted to control said video displaybased on said determined variable, wherein said object is a squeezableball.
 19. An exercise system comprising: a video display viewable by aperson exercising with an object resisting movement by the personexercising; at least one electro-optical sensor viewing a plurality ofpoints on said person; a computer to process information from saidsensor and determine a variable concerning said points, wherein saidcomputer is adapted to control said video display based on saiddetermined variable; and a microphone for audio input from said person.20. The exercise system of claim 19 wherein said object is a stretchablecord.
 21. An exercise system comprising: a stretchable member; a videodisplay viewable by a person exercising with said stretchable member; atleast one electro-optical sensor viewing a plurality of points on atleast one of said person and said stretchable member; and a computer toprocess information from said sensor and determine a variable concerningsaid points, wherein said computer is adapted to control said videodisplay based on said determined variable.
 22. The exercise system ofclaim 21 wherein said exercise is part of a rehabilitation regimen forsaid person.
 23. The exercise system of claim 21 wherein saidelectro-optical sensor includes a video camera.
 24. The exercise systemof claim 21 wherein said video display includes a substantially verticallong axis.
 25. The exercise system of claim 21 wherein said videodisplay provides information concerning said person's goals.
 26. Theexercise system of claim 21 wherein said video display providesinformation concerning said person's prior exercise performance.